The CA API Gateway allows you to stand up an SFTP endpoint for inbound messages to be processed. These messages are instantly processed by the gateway in memory, so throw away everything you knew about SFTP file uploads.
Understanding the conversation flow
When you connect to an SFTP server to upload a file, 5 things happen – note this reflects batch operations; interactive sessions may differ:
- User/service connects to the SFTP server specifying credentials via the command line (i.e. [email protected] – and likely using an SSH key)
- Change directories (optional)
- cd /directory/name
- stat /directory/name
- PUT filename
- put filename
- stat filename
What’s not obvious, or documented, is that each one of these functions (specifically SFTP command responses) must be explicitly implemented in your API policy.
First Things First – Create a Listener
SFTP uses the SSH2 protocol with a different shell on the gateway. The CA API Gateway implements its own SFTP handler to effectively intercept SFTP commands as requests, and allows these to be routed to policies on the gateway.
A listener must be created first, to allow this to occur. Once you create the listener, specifying the ports and supported SFTP commands, you have two options:
- Use Service Resolution to map requests to policies
- Select a policy that will handle all requests to the listener
In my testing, the first option never, ever worked. It’s easier to pretend CA never gave us the option and select a policy.
This is the easy one. By simply adding the “Require SSH Credentials” and “Require User Authentication” assertions, you can authenticate a specific user, such that the rest of the API policy will not function if you haven’t passed this simple entry requirement.
The gotcha? Well, if you connect to an SFTP listener without specifying credentials, you seemingly have a full SFTP connection. But without authenticating, it doesn’t do anything anyway (i.e. LIST/PUT commands will simply fail).
Handling the STAT command
This is the one that will give you the most agony. The STAT command is issued automatically by the SFTP client in the following scenarios
- After you change directory (cd)
- After you put a file (put)
The reason this is tricky is two fold; each time a command is issued, it generates a new request to your policy, which in turn means that the commands are stateless.
The purpose of the STAT command, at least according to the RFC959, is to return status of a file:
STATUS (STAT) This command shall cause a status response to be sent over the control connection in the form of a reply. The command may be sent during a file transfer (along with the Telnet IP and Synch signals--see the Section on FTP Commands) in which case the server will respond with the status of the operation in progress, or it may be sent between file transfers. In the latter case, the command may have an argument field. If the argument is a pathname, the command is analogous to the "list" command except that data shall be transferred over the control connection. If a partial pathname is given, the server may respond with a list of file names or attributes associated with that specification. If no argument is given, the server should return general status information about the server FTP process. This should include current values of all transfer parameters and the status of connections.
When this is issued after the “cd” command, it is to verify that the folder is valid.
When this is issued after the “put” command, it is to verify the file was uploaded.
For file PUTs this obviously creates an issue whereby
Given we are uploading files, the filename in the request can and should be returned in the response to allow the SFTP client to gracefully admit that
To obtain the status of files uploaded, it
Three options here:
- don’t use directories (instead, connect, and upload files directly to the root of the SFTP host, i.e. “/”)
- handle the directory change with a hardcoded STAT response which simply returns the directory name being called for
- handle the directory change by routing to your backend to actually return a response
Handling the PUT Command
This one almost doesn’t deserve
Dealing with XML files? Just set the content type on the listener to text/
One of my current projects follows a workflow where the file is uploaded, the file contents decrypted with a known key, scanned for viruses, base64 encoded, and then finally sent to a backend service to process the request.
Of course, this is followed by the SFTP client issuing the STAT command for the uploaded file, and an XML file list response is returned from a backend service, which the gateway translates into an RFC compliant STAT response to the SFTP client! Glorious!
Format of the STAT / LIST response
The response is in XML format, and the SFTP listener on the gateway will transform this XML policy response into a valid SFTP response to the client. Note that this can be dynamically generated by your backend system, OR, hardcoded within the policy for some specific use cases.
<?xml version="1.0" encoding="utf-16"?> <files> <file file="false" lastModified="0" name="thisIsADirectory" permissions="755" size="0" /> <file file="true" lastModified="0" name="thisIsAFile.xml" permissions="755" size="600" /> </files>