Misc. COM Components NetProbe

NetProbe 1.0 (beta). This file documents software in beta state and is subject of change without notice.

Overview, Using NetProbe, Object reference


NetProbe is a protocol independent ActiveX intended for sending and retrieving information through any protocol registered on the local system. It was tested with standard protocols like http, ftp, file and with custom protocols like res, mk (MS HTML Help protocol), alp (Active Local Pages) and others. So NetProbe is not just a tool for internet, but also universal tool able to fetch/send information through any available protocol no matter if it is a network oriented protocol (such as http) or a local processor (such as alp and mk).

Example tasks that can be done with NetProbe:

Fetching information from a CHM (HTML help file) file and embedding it into the pages generated on the WEB server (with a few lines of ASP code)
Sending requests POST, GET, PUT to remote servers and working with the returned data
Using external protocol engines as content servers in applications running on the desktop or on the WEB server. For example an application (or even ASP page running in IIS) may request ASP page to be executed by the ALP engine and do something with the result.
Fetching information from the internet. The NetProbe runs the same way as Internet Explorer and is able to fetch everything IE is able to download.
Synchronization/data exchange between machines/servers. For example an ASP page can download news from other servers (in any format) and select something to show.

NetProbe supports features required sometimes by the protocols - such as authentication, request/response headers manipulation. The requested information can be accessed as string or binary data as appropriate. Using additional controls (such as Storages and Files from the ActiveX Pack1 or a XML parser) NetProbe is able to provide all the required services for network or network-like communication and real-time data exchange. The data can be posted/sent through NetProbe in user-defined formats - for example if the application wants to post a form or upload files.

NetProbe comes in two forms - both and freethreaded. The applications are able to create and manage pools of NetProbe objects and are provided with certain synchronization services (for the freethreaded case mostly). For example an ASP application can create several NetProbe objects and keep them in Application variables. Then the ASP pages are allowed to wait/occupy a NetProbe object and use it. These features allow the application to optimize its network functions depending on the expected transfer and requests count. 

The applications that use NetProbe can set Timeout which means in this case "time limit" for the requested operations. This allows the server applications to control the time spent for the network operations and response to their clients in time. The actual timeout value depends on the system settings of the protocol used and may differ.

Using NetProbe

Internal structure. Internally NetProbe consist of two parts - the NetProbe object and a "call back" object not visible for the external applications. When created NetProbe automatically creates a thread with the "call back" object running in it. This object handles the requests and thus does the real work. So each instance of NetProbe has a thread running in the background. Is this a problem? In fact most of the Windows API functions that implement similar functionality in an easy-to-use manner use one or more threads so the NetProbe behavior is not unique.nor problem - controlling the object's life time the application has full control over the system resources used by the object. This behavior means that applications which need multiply concurrent network operations should prefer pools of NetProbe objects. 

The examples below are simplified versions of the examples packed with NetProbe. The packed samples are written in ASP and are designed for ALP (Active Local Pages) primary usage. Some of them use additional components and you will need to install them in order to use the samples. Before starting them take a look at the README.TXT file in the samples directory.

Simple usage in ASP page

Let us fetch a resource first. We need to create and configure the object and send a request.

Set np = Server.CreateObject("newObjects.utilctls.NetProbe")
If np.Request("http://server/resource") Then
  If np.ResponseMIME = "text/plain" Then
    ' Do something with it - for example display it
    Response.Write np.ResponseText
    ' Not the content type we have expected. 
  End If
  ' Fatal error
End If

The error detection

The error detection can be a problem sometimes. With HTTP or FTP everything will be ok unless a proxy server doesn't return incorrect HTTP codes, but with custom protocols everything depends on the protocol concepts and developer's idea. So you may need to test them before starting your project. Unfortunately this area is not well covered by the Windows API (URLMonikers and Asynchronous Pluggable Protocols) and the protocols have certain freedom to behave in different manner which makes impossible to implement universal error detection mechanism.

Extending the above example We can do the following:

Set np = Server.CreateObject("newObjects.utilctls.NetProbe")
If np.Request("http://server/resource") Then
  If np.ResponseMIME = "text/plain" Then
    ' Do something with it - for example display it
    Response.Write np.ResponseText
    ' Not the content type we have expected. 
  End If
  Response.Write "An error was occurred: " & np.ErrorCode & " " & np.Errortext
End If

Which will show the HTTP error occurred as it was reported in the response.

Other protocols may have no standard error codes or the abstraction layer (URLMonikers) may hide the details. So in general you cannot be sure what kind of error has been occured. Therefore if it is not too important you may limit your error handling to the minimum - ensuring first that the protocol returns errors instead of error screens for example. Let us put a page from HTML help file (CHM file) into ASP page:

Set np = Server.CreateObject("newObjects.utilctls.NetProbe")
If np.Request("mk:@MSITStore:" & Server.MapPath("sample.chm") & "::/sample.htm") Then
  content = np.ResponseText
  content = "Error reported: " & np.ErrorCode & " " & np.ErrorText
End If
Set np = nothing
Here comes the content from the CHM file
<%= content %>

Simple form post

NetProbe supports not only fetching but also sending data. The different protocols may name such actions with different names - upload, post and so on, however the process is almost identical with all of them. The important differences are the data encoding and other parameters certain protocol may need in order to accept the data correctly. Being universal NetProbe does not interfere in this process in dept. In fact many environments supply the required features or other components can be used to perform the encoding/decoding actions. Note also that many  protocols allow custom (application defined) ways of sending data which makes the possible forms almost infinite. The user's opinion will be interesting for us and we may add some components in the DLL (or in another DLL) for the most popular scenarios in future.

So let us post a typical HTML form, but without HTML and a browser. Suppose we want to send values named "A" and "B":

Set np = Server.CreateObject("newObjects.utilctls.NetProbe")
np.RequestMethod = "POST"
np.PostData = "A=Value1&B=Value2"
np.FormPost = True
If np.Request(http://server/page.asp) Then
  If np.ResponseMIME <> "text/plain" Then
    content = "The page returned unexpected content type: " & np.ResponseMIME
    content = np.ResponseText
  End If
  content = "Error reported: " & np.ErrorCode & " " & np.ErrorText
End If

Note that we set the RequestMethod to "POST" which will be probably expected by the page we refer on the remote server and also note that we set the FormPost property to True which ensures that the request will be tuned correctly for the FORM post case. NetProbe supports some hidden features related to the popular request methods. They are implemented in order to solve the problems caused by windows API for Asynchronous Pluggable Protocols and are able to do it not only with the HTTP protocol but also with any protocol that tries to resemble HTTP. So setting FormPost to True instructs NetProbe to do this automatically.

Another interesting piece of code is the encoding. The PostData is a string in this case and it must be URL encoded. In this simple situation we do not need any encoding because we send simple ASCII data but in practice you will need to do something more. Let us concentrate over the encoding then - suppose the values of A and B come from some variable values - vA and vB. Then we will need to prepare the PostData string as follows:

' The other code is omitted for brevity
Dim postString
postString = "A="
postString = postString & Server.URLEncode(vA) & "&"
postString = postString & "B="
postString = postString & Server.URLEncode(vB)
np.PostData = postString
' The other code is omitted for brevity

If the variable names may vary you will need to perform URL encoding over them too - to ensure they will not contain characters that will damage the post string.

Advanced data send/receive scenarios

Send a file

The standard HTTP file upload is based on a standard that is too complex for some scenarios and does not worth the efforts needed. Also if you need to send a file from one server to another there is no browser around and there is no need to follow any such a standard if everything depends on you (Note that the same can be done with ALP from the client machine to the server). So let us do it in the simplest way. Note that the sample uses the Storages and Files components from the newObjects ActiveX Pack1 - you will need to install them on the both ends.

So on the client side (which sends the file) we will have something like this:

Set np = Server.CreateObject("newObjects.utilctls.NetProbe")
np.RequestMethod = "POST"
Set sf = Server.CreateObject("newObjects.utilctls.SFMain")
Set f = sf.OpenFile(Server.MapPath("README.TXT"))
np.PostData = f.ReadBin(f.Size)
If np.Request(theURL) Then
  If np.ResponseMIME <> "text/plain" Then
    result = "The page returned unexpected content type: " & np.ResponseMIME & vbCrLf & _
             "And the following content: " & vbCrLf & Server.HTMLEncode(np.ResponseText)
    result = np.ResponseText
  End If
  result = "Error reported: " & np.ErrorCode & " " & np.ErrorText
End If

We expect plain text response from the page on the server in this example and if the content type is not what we expect probably the URL is incorrect or the page placed there is not our page.

On the server side which receives the file we have:

    Response.ContentType = "text/plain"
    If Request.ServerVariables("REQUEST_METHOD") = "POST" Then
        Set sf = Server.CreateObject("newObjects.utilctls.SFMain")
        Set f = sf.CreateFile(Server.MapPath("README-RECEIVED.TXT"))
        size = Request.TotalBytes
        f.WriteBin Request.BinaryRead(size)
        Set f = Nothing ' Close the file
        Response.Write "Success: File has been saved as README-RECEIVED.TXT (Totalbytes: " & size & ")"
        Response.Write "Error: Incorrect request method"
    End IF

The process is simple - the "client" page reads the file and posts it to the "server" page which reads it with the Request.BiaryRead ASP method and saves it to the disk - no need of anything else. In real situation we will need to add some more precautions for wrong uploads and incorrect requests, but this will not be anything hard to do - may be no more than a few lines of code. Comparing this to typical HTTP upload from a form it is many times simpler even with some upload helper components.

The Timeout property value could be important in this case. It defaults to 60s which gives the NetProbe time limit of 60 seconds to complete the task. The above example is in an ASP page and it must complete for no more than a few seconds and show something to the user - even an error will be better than long timeout. Therefore in a real case we can use the Timeout property to limit the time interval allowed for the operation and show timeout error if the operation takes too much time. If the files sent are always small a small timeout value will be a good idea, but in case of bigger files (1Mb for example) we will need to give the NetProbe chance to complete its work and set a timeout greater than 60 seconds. One simple rule to determine the time limit could be defined as the lower connection speed expected on the client machines multiplied by 2 times the size of the maximum size of the data being sent.

Consider the fact that Timeout in NetProbe means time limit! This is more suitable to the NetProbe purposes than anything else because NetProbe must not hang forever no matter if the operation continues successfully but too slowly.

These basic techniques just show the basics. Much more can be done with NetProbe combining its power with other components. In fact our aim is to give you universal mechanism to transfer data - and everything else depends on your needs. The data sent can be processed on the "server" side by some XML parsers for example or you may want to implement generic data exchange mechanism and parse the incoming data as some settings in format you defined previously. You are not limited to any standards nor to any protocols - if certain protocol supports certain feature you can use it as like any other protocol with NetProbe - one good example is ALP - it behaves like HTTP but works on your local machine without WEB server or network requirements. In some cases you may want to ask it to do some work for you and use the result. Ok, there is no problem post/send some data to it and then get the result and use it in your application.

One of the most interesting areas of NetProbe usage are generic implementations of mechanisms similar to the WEB services. Having data transfer under control you are able to implement any kind of communication through any protocol. If you do not need to follow the standard or part of the environments you support do not have support for it - well, you can do what you need yourself or even implement SOAP over NetProbe (we will do this in near future). 

What about synchronization?

n ASP for example creating a NetProbe and releasing it in each ASP page that needs it will be good enough if this happens from time to rime but if the usage of NetProbe is too frequent optimization will be required. Here come the features designed to help the developers to reuse the NetProbe objects. We will demonstrate the usage of Borrow/Free methods pair.

If Not IsObject(Application("NP")) Then
  Set Application("NP") = Server.CreateObject("newObjects.utilctls.NetProbe.free")
End If

We ensure the Application("NP") holds one free-threaded NetProbe object. This can be done also in global.asa, of course but we prefer to pack the sample in one file. Now a page must Borrow, use the object and finally Free it

If Not Application("NP").Borrow(True) Then Err.Raise 1,"Pool sample", "Cannot borrow the NetProbe object"
If Application("NP").Request("some_URL") Then
  content = Application("NP").ResponseText
  content = "Error reported: " & Application("NP").ErrorCode & " " & Application("NP").ErrorText
End If

Of course this is the most simple form of pooling- just one object is used in the entire application. More functional usage will involve also the IsBusy property and a function that will wait/return the first free NetProbe. 

In most cases the technique explained above is not needed. But suppose you have applications, may be not ASP or combined ones, that need to transfer data between each other too frequently. Then you will need to total control over the network overhead they cause and memory consumed. Using the Borrow/Free/Busy methods you can design the appropriate behavior without need of additional software (such as Microsoft Transaction Server/COM+ services).

Object reference

Threading model: both
ProgID: newObjects.utilctls.NetProbe
ClassID: 873097F5-C1D1-4142-9713-0295DD374D44

Threading model: free
ProgID: newObjects.utilctls.NetProbe.free
ClassID: 329C3A26-23A2-405c-A93D-63D7A38EB0E3

Request definition members
RequestMethod (string) Request method name. This is protocol specific, however most protocols support "GET", "POST", "PUT" methods or part of them. The request methods are defined initially in the HTTP specification but the windows URLMoniker API treats them as abstract scheme often used by the custom protocols (such as res, mk, alp). "GET" method is default (empty string will cause GET method to be used).

Remarks: Note that most of the custom protocols do not need any specific request method (e.g. the request method has no meaning - for example protocols like res or mk). Protocols that simulate WEB server behavior (like alp) support also POST method.

Username (string) User name to be used if authentication is required.
Password (string) Password to be used  if authentication is required.

Remarks: Authentication is usually required by the internet related protocols like http and ftp. It is rarely implemented by the custom protocols that work with local resources in a network like manner.

RequestHeaders (string) Additional headers application needs to send with the request. Each header must end with <new-line> (including the last).

Remarks: Part of the headers are automatically generated. If the application needs to replace some of them it should put it in this property and the application defined header will override the auto-generated header. For example this is useful if you want to specify specific custom MIME type for the post data or change the user agent string.

Note that headers have meaning only with the HTTP and HTTPS protocols. Unfortunately the standard defined by the corresponding Windows APIs does not include headers for the other protocols. 

RequestMIME (string) MIME type of the posted data. This property is a helper - the same can be done by adding custom Content-Type header in the RequestHeaders property.

Remarks: As it is mentioned for RequestHeaders property this feature is functional only for HTTP/HTTPS protocols because of the API limitations. 

PostData (variant) The data to be sent. The data must be string or binary data. As in all the popular environments and components binary data is VT_UI1 | VT_ARRAY (compatible with ASP in IIS/ALP with ADO, Storages and Files and so on).

Remarks: NetProbe will not perform any encoding over the binary data - it must be encoded and prepared by the application. There are many standards and ways to send data. To determine if you need any encoding and which one review the protocol documentations and consider the server application capabilities. For example your own ASP page on a server may expect raw non-encoded data in which case you will not need to do anything else than putting the data in this property. And contrary if you want to send data to an ASP page that expects file upload, for example, you will need to take care to pack the data using the multipart/form-data encoding. Therefore encoding depends on the "other side" and you need to prepare your data in the appropriate form for it - and if the both sides are yours you are able to make them simple if the internet standards are not important in this particular case.

If the data is a string it is converted or not converted depending on the unicodePost property and the codePage property. Usually the text data should be sent as ANSI text, but in case of custom application you can force the NetProbe to send unicode text by setting the unicodePost property to True.

If you want to post a form you will need to set an URL encoded set of variables in this property. For example a form containing 3 fields F1, F2, F3 with some values - "val1", "val 2", "val 3" should be posted as:


Many environments have built-in URL encode routines (IIS, ALP) and the application may use them to encode the posted data.  

FormPost (Boolean) A helper property that helps you to configure correctly typical form POST request. If set to true property will set a Content-Type header to application/x-www-form-urlencoded and will tune also some other parameters. In other words if you want to simulate posting a form from a HTML page set this property to True.

Remarks: IIS requires this content type to be set in order to serve the parameters send in the PostData in the Request.Form collection to the requested ASP page.

If set to true this property will override the formUpload property.

Note that there is no clear mechanism to recognize the "meaning" of the post for the custom asynchronous pluggable protocols (non-internet related). This property tunes the binding operation to behave in manner similar to the HTTP protocol and thus most custom protocols will successfully recognize the post as typical form post.

formUpload (Boolean) Like the FormPost property but tunes the request for multipart/form-data encoded forms. These are the forms used for file uploads. If you want to send request to a page/server which expects file upload form set this property to True.
unicodePost (Boolean) Sends string PostData as unicode text. 

Remarks: Most protocols expect ANSI text, but if you are going to use some ASP pages or CGI scripts with NetProbe only you can design the requesting application to send unicode text and thus skip any encoding problems.

codePage (long) A code page value. If the unicodePost is False any string posted through the PostData property will be encoded using this code page.
Timeout (long) Time limit in seconds. Default is 60. This property sets a time limit for the operation. No matter if the operation is successful or not it will be cancelled if the time limit is exceeded.

Remarks: NetProbe is intended for many different purposes including usage on WEB servers. So time limit is an important feature that allows the application to specify absolute maximum of the time permitted for the request. This will allow an ASP page, for example, to complete in time if the operation is too slow due to network problems or some other reason.

The connection timeout cannot be controlled through NetProbe. The different protocols have different specifics - some are synchronous, others are network related and a "connection timeout" setting may have or may not have any meaning with certain protocol. If you want to relay on the protocol settings just set a big value to the Timeout property (such as 80000 - about one day).

Perform request
variable = Request(url) This method sends/performs the request.
url is a string that is the requested full URL - without any bookmark information (#anchorname).
returned value - Boolean. The result of the operation - True means success, False failure/timeout.

The url parameter may include parameters (?parameters form).

Remarks: In synchronous mode (default) the method will not return until the request is completed (successfully or not). In asynchronous mode the method will return immediately and the application will need to sink the NetProbe events in order to determine the request progress and completion.

One NetProbe instance can handle only one request at a time. The method will return an error if another request is started but not yet completed. 

Asynch (Boolean) Request mode. Default is False (synchronous). See the Request method for details.
CancelRequest Cancels any request in action. (Can be called from any thread)
Access the response
ResponseText(bUnicode, lCP) (string) The response as text. The application should determine if the response is text by inspecting the ResponseMIME property.
bUnicode - Boolean (optional, default is False). The response data will be treated as unicode text.
lCP - long (optional, defaults to ANSI code page). If the bUnicode is False (default) this code page will be used to convert the response data to string.

Remarks: As like the Response property this property can be read many times. The response data is not deleted/released after reading the property. To clean the last request - call CancelRequest.

Response (binary data) The response as binary data (VT_UI1 | VT_ARRAY). See also ResponseText.

Remarks: The application may read the response as text and as binary data and as both if this can be of some use.

ResponseMIME (string) Response MIME type as reported. If nothing is reported by the protocol the property will return application/octet-stream.
ResponseHeaders (string) Any headers reported with the response.

Remarks: This property has a meaning only with HTTP and HTTPS protocols.

ErrorCode (long) The result of the operation. 0 - means success, any other value is an error code.

Value of (-1) is reserved for time out error.  Other values are protocol specific.

Remarks: Note that NetProbe does support automatic mechanism for error recognition only for standard internet protocols such as HTTP/HTTPS, FTP and so on. In general the Asynchronous Pluggable Protocols are different - some may prefer to return success and prepare an error page for the user instead of indicating the error by returning a failure code. Depending on their design the error detection will require different logic and sometimes will be possible only by inspecting the returned content. This may happen also with HTTP protocol - if you are using proxy server that returns success and user friendly error page (most commercial proxies work correctly and this will not be a problem). Also even if a HTTP proxy returns an error code it could be different than the real HTTP error code.

When working with custom protocols it is strongly recommended to review their documentation and write some test scripts to check the returned information and find out how the errors can be detected. In general service/supplementary protocols such as res: or mk: are designed for usage in other applications in the background and will be very easy to use but others may target the users directly (designed to show HTML pages for example) and may prefer to never return an error that can be automatically detected.

ErrorText (string) Supplied by some protocols and the NetProbe itself if the error is generated internaly.
Synchronization (these methods are meaningful mostly if the freethreaded version is used)
IsBusy (Boolean) Returns True if a request is in progress and False otherwise.
variable = Borrow(bWait) These two methods are optional. They may help an application to maintain pools of NetProbe objects.

Borrow - requests ownership. If bWait is set to True the method will wait for up to 60 seconds the object to be freed by its other clients. If bWait is False the method will return immediatelly.
The return result is Boolean - True if the ownership is obtained and False if not. 

Free - frees the object.

Remarks: These two methods are just helpers. They allow the applications to implement in a simple manner custom object pooling logic. For example an ASP application may construct several NetProbe objects and save them into the ASP Application object's collection and then use the first free (not owned) object available. This technique is not needed if the application needs only a few requests and only in some rare situations but in case of high traffic implementing pooling mechanisms may give the application much more opportunities.

OnOpenRequest Fired when the request starts
OnFinishRequest(bSuccess) Fired when the request finishes. bSuccess - is True if the operation completed successfuly, and False if the operation failed.
OnProgress(progress, max) Fired during the operation to indicate the progress.
progress - (long number) current progress
max - maximum amount of work expected

Remarks: Note that many protocol handlers report incorrect or inaccurate progress values. 

OnProgressInfo(nType, infoText) Fired with the OnProgress event.
nType - code (long) - information type.
infoText - string - an information text.
newObjects Copyright 2001-2006 newObjects [ ]