The response buffers are used and returned to the host in the same order that they were given to the INIC. The valid field is set by the INIC before returning the buffer to the host. In this way a PCI interrupt, with a single bit in the interrupt register, may be generated to indicate that there is a response buffer for the host to process.
When servicing this interrupt, the host will look at its queue of response buffers, reading the valid field to determine how many response buffers are to be processed. It is designed that the setting of any bits in the ISR will cause an interrupt, provided the corresponding bit in an Interrupt Mask Register is set. The default setting for the IMR is 0. For the host to never have to actually read the register from the INIC itself, it is important for the INIC to update this host copy of the register whenever anything in it changes.
The host will Ack or deassert events in the register by writing to the register with 0's in appropriate bit fields. So that the host does not miss events, the following scheme is employed:. Then the INIC starts accumulating any new events not reflected in the host copy in a separate word. This is called NEWA. Table 1 lists the INIC register addresses. For the sake of simplicity, the registers are in 4-byte increments from whatever the TBD base address is.
The ATCP driver runs on the host and consists of three main components. In order to ensure that our ATCP driver is written in a consistent manner, we have adopted a set of coding guidelines. These proposed guidelines were introduced with the philosophy that we should write code in a Microsoft style since we are introducing an NT-based product.
The guidelines below apply to all code that we introduced into our driver. We adhere to this style. Structure pointers—Microsoft typedefs all of their structures. Function calls—Microsoft separates function call arguments on separate lines:. Function comments—Microsoft includes comments with each function that describe the function, its arguments, and its return value.
We also include these comments, but move them from within the function itself to just prior to the function for better readability. These keywords denote whether the function argument is used as an input parameter, or alternatively as a placeholder for an output parameter.
We include these keywords. Prototypes of existing BSD functions are left in their current header files, however, to minimize differences between our code and the BSD base. Indentation—Microsoft code fairly consistently uses a tabstop of 4.
For example, foo. The basic rule for SMP kernel code is that any access to a memory variable must be protected by a lock, which prevents a competing access by code running on another processor. Spinlocks are the normal locking method for code paths that do not take a long time to execute and which do not sleep.
In general each instance of a structure includes a spinlock, which must be acquired before members of that structure are accessed, and held while a function is accessing that instance of the structure. In addition, every global data structure such as a list or hash table also has a protecting spinlock which must be held while the structure is being accessed or modified.
Existing list manipulations in the FreeBSD code will be left as-is to minimize code disturbance, except of course that the necessary spinlock acquisition and release must be added around them. Spinlocks should not be held for long periods of time, and most especially, must not be held during a sleep, since this will lead to deadlocks.
There is a significant deficiency in the NT kernel support for SMP systems: it does not provide an operation, which allows a spinlock to be exchanged atomically for a sleep lock. This would be a serious problem in a UNIX environment where much of the processing occurs in the context of the user process, which initiated the operation.
The spinlock would have to be explicitly released, followed by a separate acquisition of the sleep lock: creating an unsafe window. The calling thread does NOT sleep at that point: it returns, and may go on with other processing.
Thus we that have not in fact used sleep locks anywhere in the design of the ATCP driver, so hopefully the above issue will not arise. As described above, the ATCP driver supports two paths for sending and receiving data, the fast-path and the slow-path. For a first implementation, we divide network communication into NETBIOS traffic, which is identifiable by port number, and everything else. Alternatively the host can acquire this information from the header itself, but since the INIC has already done the decode, it seem reasonable to just pass it.
From the TDI spec, the amount of data in the buffer actually sent must be at least bytes. For segments less than a full byte payload, all of the received segment will be forwarded; it will be absorbed directly by the TDI client without any further memory descriptor list MDL exchange.
Note that the INIC maintains its advertised receive window as the maximum currently bytes while filling the MDL, to maximize throughput from the client.
At this point the cycle of events is complete, and the ATCP driver is now waiting for the next header indication. In the general case we do not have a higher-level protocol header to enable us to predict that more data is coming. The original idea was to accumulate segments until a given amount e.
A problem with this approach is that the INIC would be required to close its advertised receive window as segments were accumulated, which would stall output from the sending client. To avoid this, we resorted after some experimentation to a subterfuge. However, for small requests, there is no MDL returned by the TDI client: it absorbs all of the data directly in the receive callback function.
In this case we need to update the INIC's view of data which has been accepted, so that it can update its receive window. In order to be able to do this, the ATCP driver accumulates a count of data which has been accepted by the TDI client receive callback function for a connection.
We therefore piggyback the update on requests sent out to the INIC. Whenever the ATCP driver has outgoing data for that connection, it places this count in a field in the send request and then clears the counter.
The fast-path transmit or output data flow is considerably simpler. The INIC will copy the data from the given physical location s as it sends the corresponding network frames onto the network. Note that there may be multiple output requests pending at any given time. FTP often use double-buffering. In the outgoing direction, we have protocol-allocated MBUFs in which to assemble the data and headers.
Most of the functions in this category are in the ATCP file atktdi. The latter may complete back to its caller as soon as the data has been copied into the socket buffer. The criterion is basically that the connection is on an interesting protocol port, and is currently quiescent: i. The problem arises with incoming slow-path data. If we attempt to do the context-pass in a single command handshake, there is a window during which the ATCP driver has sent the context command, but the INIC has not yet acquired or has not yet completed setting up the context.
During this time, slow-path input data frames could arrive and be fed into the slow-path ATCP processing code. We could simply abort the outward pass of the context in this event, but it turns out that this scenario is quite common.
So it seems better to have a reliable handshake, which is accomplished with a two-exchange handshake. Receipt of the response to this initial command does not suffice to provide a reliable interlock, however.
It is possible that when the response to the initial command is received, there are still slow-path frames in a queue waiting to be delivered. This frame identifies the CCB context and signals that no further slow-path frames will follow for that context.
Since this frame travels on the same pathway as data frames, we know when we receive it that it signifies the end of any possible slow-path data. If a flush situation arises while a migration command is outstanding, the condition is noted in host connection flags, and the actual flush command is sent only when the NULL frame is received in the first-half case or the command response is received in the second-half case.
The INIC sends an error response to any current outstanding receive request it is working on corresponding to an MDL into which data is being placed. Likewise the INIC sends an error response for any outstanding send requests. Note that part of the information provided with a context is the address of the CCB in the host.
Sending this indication via the regular input path ensures that it will arrive before any following slow-path frames. At this point, the INIC is no longer doing fast-path processing. It discards its CCB context for this connection, and any further incoming frames for the connection will simply be sent to the host as raw frames for the slow input path. As soon as the ATCP driver detects that a flush is in progress on a connection, it sets a state flag on its connection context to indicate this fact.
The ATCP driver may become alerted about a flush in several ways: it might be an explicit host-initiated flush, or it may see either the flush frame or an error on a send or receive request. The order in which these are received may vary because, as we noted earlier, the receive frame and command response paths are unrelated. This involves completing any IRPs corresponding to requests which have entirely completed, adjusting fields in partially-completed requests so that send and receive of slow path data will resume at the right point in the byte streams and propagating any timer expiration states from the INIC to the BSD code.
The following paragraphs discuss the issues associated with porting this code, the FreeBSD code itself, and the modifications required for it to suit our needs. Note, however, that details of the higher, TCP-level part of the port are postponed until later, since this needs some groundwork from the discussion of the NT TDI interface. It contains code to handle a variety of interface types and many different kinds of protocols. To meet this requirement the code is often written in a sometimes confusing, convoluted manner.
General-purpose structures are overlaid with other interface-specific structures so that different interface types can coexist using the same general-purpose code. For our purposes much of this complexity is unnecessary since we are initially only supporting several specific protocols. It is therefore tempting to modify the code and data structures in an effort to make it more readable, and perhaps a bit more efficient. There are, however, some problems with doing this. For this reason we have initially kept the data structures and code at close to the original FreeBSD implementation as possible.
The code has, however, been modified for several reasons. The interface of this code to the NT system requires some significant code modifications. Modifications for SMP are also necessary. Further, unnecessary code has been removed. These include bcopy to copy memory, malloc to allocate memory, timestamp functions, etc. These will not be itemized in detail since the conversion to the corresponding NT calls is a fairly trivial and mechanical operation.
Under FreeBSD, network buffers are mapped using mbufs. Under NT network buffers are mapped using a combination of packet descriptors and buffer descriptors the buffer descriptors are really MDLs. There are a couple of problems with the NT method. First it does not provide the necessary fields which allow us to easily strip off protocol headers. Second, converting all of the FreeBSD protocol code to speak in terms of buffer descriptors is an unnecessary amount of overhead.
In addition each mbuf and will point to a packet descriptor which is associated with the data being mapped. Once an NT packet is mapped, our transport driver should never have to refer to the packet or buffer descriptors and for any information except when we are finished and are preparing to return the packet.
There are a couple of things to note here. The INIC has been designed such that a packet header should never be split across multiple buffers. One such example of this is ARP frames. Under these circumstances, it is important that we do not modify the data, or the packet and buffer descriptors.
We will discuss this further below. Also note that we allocate a pool of mbuf headers at ATCP initialization time. The reason for this is that we will be receiving data from the card that has already been acknowledged by TCP.
Because of this it is important that we never run out of mbuf headers. To solve this problem we statically allocate mbuf headers for the maximum number of buffers that we will ever allow to be outstanding.
By doing so, the card will run out of buffers in which to put the data before we will run out of mbufs, and as a result, the card will be forced to drop data at the link layer instead of us dropping it at the transport layer. We also use a pool of actual mbufs not just headers. These mbufs are needed in order to build output packets for the slow-path data path, as well as other miscellaneous purposes such as for building ARP requests.
We allocate a pool of these at initialization time and add to this pool dynamically as needed. Unlike the mbuf headers described above, which are used to map acknowledged TCP data coming from the card, the full mbufs contain data that can be dropped if we cannot get an mbuf.
There are a variety of structures, which represent a single interface in FreeBSD. As mentioned in the porting philosophy section, many of the above structures could likely be collapsed into fewer structures. In order to avoid making unnecessary modifications to FreeBSD, for the time being we have these structures mostly unchanged. We have, however, eliminated the fields from the structure that will never be used.
These structure modifications are discussed below. We also show in FIG. This is a structure that we define, in proto. It contains the arpcom structure, which in turn contains the ifnet structure.
FreeBSD initializes the above structures in two phases. Initialization still occurs in two phases, but the details are different:. For each one, it allocates an IFACE structure, and does further registry scanning to obtain parameters for this interface. Note that so far, everything has been done from information in the registry; we do not yet have any contact with physical hardware. It then does a number of other queries for characteristics of the interface and stores the results in the IFACE.
Additionally, we need to flush any fast-path connections using this interface back to the host; this is done by the ATKIfRouteFlush function in atkfastpath.
While we do not change the functionality of this file, we rename it to a more logical arp. These structures do not require major modifications. The functions that require modification are defined here. This introduces several scenarios for an incoming ARP frame. It does this in anticipation of the fact that any host that wishes to know our MAC address is likely to wish to talk to us soon. As mentioned above, since ARP requests are broadcast, we see every one on the network. When we receive an ARP request of this type, we simply check to see if we have an entry for the host that sent the request in our ARP cache.
If we do, we check to see if we still have the correct MAC address associated with that host. If it is incorrect, we update our ARP cache entry. Note that we do not create a new ARP cache entry in this case. Having resolved the address, we check to see if there is any transmit requests pending for the resolve IP address, and if so, transmit them. Arpwhohas is a single line function that serves only as a wrapper around arprequest.
We remove it and replace all calls to it with direct calls to arprequest. Arprequest simply allocates a mbuf, fills it in with an ARP header, and then passes it down to the ethernet output routine to be transmitted.
For us, the code remains essentially the same except for the obvious changes related to how we allocate a network buffer, and how we send the filled in request. We simply move this functionality into the interface initialization code and remove this function. Arptimer is a timer-based function that is called every 5 minutes to walk through the ARP table looking for entries that have timed out.
Although the time-out period for FreeBSD is 20 minutes, RFC does not specify any timer requirements with regard to ARP so we can modify this value or delete the timer altogether to suit our needs. Either way the function doesn't require any major changes. On first thought, it might seem that we have no need for routing support since our ATCP driver will only receive IP datagrams whose destination IP address matches that of one of our own interfaces.
We do, however, need to maintain an up-to-date routing table so that we know a whether an outgoing connection belongs to one of our interfaces, b to which interface it belongs, and c what the first-hop IP address gateway is if the destination is not on the local network.
We discuss four aspects on the subject of routing in this section. They are as follows: 1 The mechanics of how routing information is stored, 2 The manner in which routes are added or deleted from the route table, 3 When and how route information is retrieved from the route table, 4 Notification of route table changes to interested parties. This is a complicated algorithm that is a bit costly to set up, but is very efficient to reference.
Since the routing table should contain the same information for both NT and FreeBSD, and since the key used to search for an entry in the routing table will be the same for each the destination IP address , we port the routing table software to NT without any major changes.
This file is ported directly to the ATCP driver with insignificant changes. Routes can be added or deleted in a number of different ways. The kernel adds or deletes routes when the state of an interface changes or when an ICMP redirect is received. User space programs such as the RIP daemon, or the route command also modify the route table. For kernel-based route changes, the changes can be made by a direct call to the routing software.
The FreeBSD software that is responsible for the modification of route table entries is found in route. The primary routine for all route table changes is called rtrequest. Other routines in the route. All of these routines found in route. For user-space-based changes, we will have to be a bit more clever. In FreeBSD, route changes are sent down to the kernel from user-space applications via a special route socket.
This code is found in the FreeBSD file, rtsock. Obviously this will not work for our ATCP driver. Instead the filter driver portion of our driver will intercept route changes destined for the Microsoft TCP driver and will apply those modifications to our own route table via the rtrequest routine described above.
Obviously, none of the code from rtsock. This same procedure will be used to intercept and process explicit ARP cache modifications. While we will keep this basic operation as is, we require a slight modification to allow us to coexist with the Microsoft TCP driver.
When an active connection is being set up, our filter driver has to determine whether the connection is going to be handled by one of the INIC interfaces.
To do this, we consult the route table from the filter driver portion of our driver. This is done via a call to the rtalloc1 function found in route. When a route table entry changes, there may be connections that have pointers to a stale route table entry. These connections need to be notified of the new route.
If the entry is no longer valid, its reference to the stale route table entry is removed, and an attempt is made to allocate a new route to the destination. For the slow-path, this works fine. Unfortunately, since our IP processing is handled by the INIC for the fast-path, this sanity check method will not be sufficient. Instead, we will need to perform a review of all of our fast path connections during every route table modification.
This picks up the new route, and uses this to build a new template when the connection is later handed out to the INIC again. Of the 10 possible ICMP message types, there are only three that we need to support. The following paragraphs describe how we handle these ICMP frames. First, it causes the route table to be updated with the route given in the redirect. As mentioned in the Route section above, we also require a call to a routine which reviews all of the TCP fast-path connections, and flushes any using the affected route.
We do the same. A source quench is sent to cause a TCP sender to close its congestion window to a single segment, thereby putting the sender into slow-start mode. We keep the FreeBSD code as-is for slow-path connections. For fast path connections we must flush the context back to the host, since we are not, at least at the moment, handling congestion and slow-start on the INIC.
These structures contain all the information needed by IP to be able to pass incoming data to the correct protocol above it. We strip the protosw array to exclude unnecessary protocols. First, since we only handle datagrams for which we are the final destination, we are never required to forward an IP datagram. IP options supported by FreeBSD at this time include record route, strict and loose source and record route, and timestamp. Since we will not be forwarding IP datagrams, this seems to be of little use to us.
It can do this because it no longer requires the original IP header. This is an absolute no-no with the NDIS 4. This is not the only place in which the FreeBSD code modifies the contents of a network buffer. It also does this when performing endian conversions. At the moment we leave this code as is and violate the DDK rules. We can do this because we ensure that no other transport driver looks at these frames.
If this changes, we can modify this code substantially by moving the IP reassembly fields into the mbuf header. Regarding IP output, only two modifications are made. The first is that since, for the moment, we are not dealing with IP options, there is no need for the code that inserts the IP options into the IP header.
Second, we may discover that it is impossible for us to ever receive an output request that requires fragmentation. These routines are limited to those that are called indirectly by the INIC miniport driver beneath us. The NDIS protocol driver initialization occurs in two phases. We attempt to allocate many of the required resources as soon as possible so that we are more likely to get the memory we want.
This mostly applies to allocating and initializing our mbuf and mbuf header. We call NdisRegisterProtocol to register our set of protocol driver routines. This completes the interface initialization. Before we describe this routine, consider each possible receive type and how it will be handled.
By binding the drivers in this fashion, we can choose to direct incoming network data to our own ATCP transport driver, the Microsoft TCP driver, or both. So, to deliver an incoming packet to our driver, we simply map the data such that byte 12 contains a non-recognized ethernet type field. Note that we choose a value that is greater than bytes so that the transport drivers do not confuse it with an We also choose a value that will not be accepted by other transport driver such as Appletalk or IPX.
Note that since we will also see these frames we can choose to accept or not-accept them as necessary. This includes both slow-path frames and fast-path frames. In the slow-path case, the TCP frames are given in their entirety headers included. If in the above example, there is an IP address This means that we need to know every IP address in the system and filter frames based on the destination IP address in a given TCP datagram.
This is done in the INIC miniport driver. One such example is a fast path receive in which the ATCP driver needs to be notified of how much data the card has buffered. To accomplish this, the first and sometimes only buffer in a received packet will actually be a INIC header buffer. The header buffer contains status information about the receive packet, and may or may not contain network data as well.
The ATCP driver recognizes a header buffer by mapping it to an ethernet frame and inspecting the type field found in byte These protocol drivers can choose to accept or not accept the data. They can either accept the data by copying the data out of the packet indicated to it, or alternatively they can keep the packet and return it later via a call to NdisReturnPackets.
By implementing it in this fashion, NDIS allows more than one protocol driver to accept a given packet. For this reason, when a packet is delivered to a protocol driver, the contents of the packet descriptor, buffer descriptors and data must all be treated as read-only.
At the moment, we violate this rule. We choose to violate this because much of the FreeBSD code modifies the packet headers as it examines them mostly for endian conversion purposes. Rather than modify all of the FreeBSD code, we will instead ensure that no other transport driver accepts the data by making sure that the ethernet type field is unique to us no one else will want it. Obviously this only works with data that is only delivered to our ATCP driver.
While this is less efficient than keeping the data and returning it later, ARP and ICMP traffic should be small enough, and infrequent enough, that it doesn't matter. This requires that the call to NdisReturnPackets must occur in a different execution context. We accomplish this by scheduling a DPC, or alternatively scheduling a system thread, or scheduling a kernel thread of our own. A DPC requires a queue of pending receive buffers on which to place and fetch receive packets.
When we finish with the data we will need to figure out whether or not to return the data to NDIS or not. This will be done via fields in the mbuf header used to map the data. When the mfreem routine is called to free a chain of mbufs, the fields in the mbuf will be checked and, if required, the packet descriptor pointed to by the mbuf is returned to NDIS.
As noted in the section on mbufs above, we map incoming data to mbufs so that our FreeBSD port requires fewer modifications. Depending on the type of data received, this mapping will appear differently. In this example, the TCP data is fully contained in a header buffer The header buffer is mapped by the mbuf and sent upstream for fast-path TCP processing.
In this case it is required that the header buffer be mapped and sent upstream because the fast-path TCP code needs information contained in the header buffer in order to perform the processing. When the mbuf in this example is freed, the mfreem routine will determine that the mbuf maps a packet that is owned by NDIS and will then free the mbuf header only and call NdisReturnPackets to free the data. In FIG. In this example the mbuf points to the start of the TCP data directly instead of to a header buffer Since a data buffer will be sent up for slow-path FreeBSD processing, we cannot have the mbuf pointing to the header buffer FreeBSD would get awfully confused.
Again, when mfreem is called to free the mbuf, it will discover the mapped packet, free the mbuf header, and call NDIS to free the packet and return the underlying buffers. Note that even though we do not directly map the header buffer with the mbuf we do not lose it because of the link from the packet descriptor. Note also that we could alternatively have the INIC miniport driver only pass us the TCP data buffer when it receives a slow-path receive.
This would work fine except that we have determined that even in the case of slow-path connections we are going to attempt to offer some assistance to the host TCP driver most likely by checksum processing only. Leaving the header buffer connected seems the most logical way to do this. In this case the mbuf simply points to our data , with no corresponding packet descriptor.
When we free this mbuf, mfreem will discover this and free not only the mbuf header, but the data as well. This receive mechanism may also be used for other purposes besides the reception of network data.
It does this by filling in a header buffer with appropriate status and delivering it to the INIC driver. The INIC driver in turn delivers it to the protocol driver which will treat it essentially like a fast-path TCP connection by mapping the header buffer with an mbuf header and delivering it to TCP for fast-path processing.
There are two advantages to communicating in this manner. First, it is already an established path, so no extra coding or testing is required. Second, since a context flush comes in, in the same manner as received frames, it will prevent us from getting a slow-path frame before the context has been flushed.
Incoming data is mapped to an ethernet frame and the type field is checked. If the type field contains our custom INIC type TCP for example , and if the header buffer specifies a fast-path connection, allocate one or more mbufs headers to map the header and possibly data buffers. Set the packet descriptor field of the mbuf to point to the packet descriptor, set the mbuf flags appropriately, queue the mbuf, and return 1.
If the header buffer specifies a slow-path connection, allocate a single mbuf header to map the network data, set the mbuf fields to map the packet, queue the mbuf and return 1. If the type field of the frame instead indicates ARP or ICMP, a mbuf with a data buffer is allocated, the contents of the packet are copied into the mbuf, the mbuf is queued, and return 0 not accepted. The receive processing will continue when the mbufs are dequeued. It will dequeue a mbuf from the queue.
If the mbuf is meant for fast-path TCP, it will call the fast-path routine directly. Otherwise it will call the ethernet input routine for slow-path processing. Since this routine is called asynchronously, our ATCP driver must save any required context into the packet descriptor header so that the appropriate resources can be freed.
This is discussed further below. Like the Receive path described above, the Transmit path is used not only to send network data, but is also used as a communication mechanism between the host and the INIC. Some examples of the types of sends performed by the ATCP driver follow. When the ATCP driver receives a transmit request with an associated MDL from a client such as a host application, it packages up the MDL physical addresses into a command buffer , maps the command buffer with a buffer descriptor and a packet descriptor , and calls NdisSendPackets with the corresponding packet.
We allocate and use a mbuf to hold the command buffer. By doing this we can store the context necessary in order to clean up after the send completes. This context includes a pointer to the MDL as well as other connection context. The other advantage to using a mbuf to hold the command buffer is that it eliminates having another special set of code to allocate and return command buffer. We store a pointer to the mbuf in the reserved section of the packet descriptor so we can locate it when the send is complete.
As described above, the receive process typically occurs in two phases. First the INIC fills in a host receive buffer with a relatively small amount of data, but notifies the host of a large amount of pending data either through a large amount of buffered data on the card, or through a large amount of expected NetBios data.
This small amount of data is delivered to the client through the TDI interface. The client then responds with a MDL in which the data should be placed. This relationship between the MDL, command buffer and buffer and packet descriptors are the same as shown in the Fast-path send section above. At this point a command buffer will be filled with pointers to the ethernet frame, the command buffer will be mapped with a packet descriptor and a buffer descriptor and NdisSendPackets will be called to hand the packet off to the miniport.
Since we will use a mbuf to map the command buffer , we can simply link the data mbufs directly off of the command buffer mbuf. This will make the freeing of resources much simpler. First it examines the reserved area of the packet descriptor to determine what type of request has completed. Dec 19, Nov 18, Sep 21, Add a basic clang-format configuration file.
Jun 7, Jun 4, Dec 31, LOCKS: update current locks. Jun 9, Remove mips from universe. Oct 4, Dec 5, Whitespace cleanup.
Mar 12, Nov 11, View code. LIB for more information. NOTES contains documentation of all possible entries. About FreeBSD src tree read-only mirror www. Releases tags.
Packages 0 No packages published. Create your free account to continue reading. Sign Up. Like this presentation? Why not share! Embed Size px. Start on. Show related SlideShares at end. WordPress Shortcode. Next SlideShares. Download Now Download to read offline and view in fullscreen. Download Now Download Download to read offline. Bangladesh Network Operators Group Follow. Working at bdNOG. EVPN Introduction.
Routing Security - its importance and status in South Asia. Related Books Free with a 30 day trial from Scribd. The Art of War Sun Tsu.
0コメント