1

A project we work with needs some data from a third-party that is given to us through FTP, and we get the file to later process it as a part of our pipelines. It has suddenly become a problem as we can't get the data anymore. I've attempted getting the data through both active (connection hangs) and passive mode, to no success.

I've noticed that I am able to get the file when using FileZilla, so I'm sure that I could somehow reproduce what FileZilla does to get the file programmatically. The issue seems to be a configuration error on the third-party's side, as when we do requests in Passive mode, we get a local IP address from the server instead of the actual server's IP. FileZilla outputs the following:

Command:    PASV
Response:   227 Entering Passive Mode (a local IP address is given here).
Status: Server sent passive reply with unroutable address. Using server address instead.

What does FileZilla do to use the server address instead? I've tried reproducing this through manual FTP commands but haven't had any luck.

hsf
  • 13
  • 1
  • 3
  • You should be asking the third party to fix their stuff .... – DavidPostill Aug 03 '20 at 11:40
  • @DavidPostill agreed, we did, but they are notoriously slow and there's nothing we can do about it. Ideally, we'd find an alternate solution meanwhile. – hsf Aug 03 '20 at 11:43
  • Is the unroutable IP address you get always the same? What is your local OS? – Kamil Maciorowski Aug 03 '20 at 11:51
  • What do you do differently with FileZilla when you succeed getting the file? – harrymc Aug 03 '20 at 11:55
  • 1
    @harrymc Nothing needs to be done differently, [Filezilla tries to use the routable IP address automatically](https://superuser.com/a/1195591/432690). Other clients may or may not save the day this way. – Kamil Maciorowski Aug 03 '20 at 12:43
  • @KamilMaciorowski: I know that, but you are right, I misunderstood the poster. – harrymc Aug 03 '20 at 12:55
  • @KamilMaciorowski the unroutable local IP address I get is always the same, only the port changes (as expected). I am running Ubuntu, the production server that runs the client is Debian. Unsure about the third-party FTP server. – hsf Aug 03 '20 at 13:24
  • Try [DNAT for outgoing connections](https://unix.stackexchange.com/a/417819/108618) and translate the unroutable IP address to the proper routable one. Remarks: (1) You may want/need to use nftables instead of iptables. (2) Do not change the port. See `man 8 iptables-extensions` for details. (3) A simple rule will DNAT all traffic destined to the IP. If the same IP address is used in your LAN then packets destined to it will leak to the remote host instead. Worse, if the same IP address is your gateway then you will(?) break your connection. (4) Not tested, therefore just a comment. – Kamil Maciorowski Aug 03 '20 at 20:15

1 Answers1

1

The server you are connecting to is badly configured.

The way the FTP handshake works is described by Wikipedia File Transfer Protocol (FTP).

  • In active mode, the client starts listening for incoming data connections from the server on port M. It sends the FTP command PORT M to inform the server on which port it is listening. The server then initiates a data channel to the client from its port 20, the FTP server data port.
  • In situations where the client is behind a firewall and unable to accept incoming TCP connections, passive mode may be used. In this mode, the client uses the control connection to send a PASV command to the server and then receives a server IP address and server port number from the server, which the client then uses to open a data connection from an arbitrary client port to the server IP address and server port number received.

The FTP server in your case most likely should be connected-to using passive mode. However, when sending its server IP address and port, it has by mistake sent its local IP address on its local network, which is of course inaccessible (or unroutable) on the outside from the internet.

FileZilla has protected itself from such cases, as described in the link given by user Kamil Maciorowski.

FileZilla simply check for IP addresses that are typically local, and ignores any such IP address sent by the server, continuing to use the initial IP address as used for the initial connection.

In short, the problem is not on your side, but on the server side, where some bad configuration was now done that confuses your third-party app. The app is probably truly trying to use this unusable IP address.

You should get in touch with both parties, the app author and the server admin, and ask them each to correct his own error. The server should return the right IP, and the app should ignore any returned local IP (or any returned IP at all).

harrymc
  • 455,459
  • 31
  • 526
  • 924
  • Thanks for the input, indeed that is the case. We are the app developers, so we can address issues on our end. We had already identified the issue but were trying to find a way around it (much in the way that FileZilla does it). In my lack of experience with the protocol, I was hoping to find out how I can use the server address instead of the given local address, but from these replies, I'm assuming that this is something that needs to be done on the TCP layer code, rather than through the FTP protocol commands. – hsf Aug 03 '20 at 13:21
  • Doing it the FileZilla means checking for typical local IPs. In the large majority of cases the returned IP can simply be ignored. It will only be different for fairly complex server architectures. The immediate patch may be to just continue using the same IP, but I do not know of course your case in detail. – harrymc Aug 03 '20 at 13:27
  • Small update: I managed to make it work by taking the same approach as FileZilla, by connecting to a socket on the same IP address that the client had been using so far. Thanks for the help – hsf Aug 03 '20 at 14:22