Jonas Stawski

Everything .NET and More

Downloading multiple files from an FTP using wildcards

Recently someone asked me how to download multiple files from an FTP using wildcards. While I started looking for an answer I found out that is not possible using the FTPWebRequest class. The good news is that although downloading using wildcards is not supported, wilcards are supported for listing files on a directory. This leaves us with one choice: first we get a list of files to download and then we download them one by one. Below is the code, I hope this helps you...

public void DownloadFiles(string WildCard)
{
 //WildCard = "*Parts.csv"
 string[] Files = GetFiles(WildCard);
 foreach (string file in Files)
 {
  DownloadFile(file);
 }
}

private string[] GetFiles(string WildCard)
{
 string ReturnStr = "";
 //Connect to the FTP
 FtpWebRequest request = WebRequest.Create("ftp://localhost/Completed/" + WildCard) as FtpWebRequest;
 //Specify we're Listing a directory
 request.Method = WebRequestMethods.Ftp.ListDirectory;
 request.Credentials = new NetworkCredential("SomeUser", SomePassword

 StringWriter sw = new StringWriter();
 //Get a reponse
 WebResponse response = request.GetResponse();
 Stream responseStream = response.GetResponseStream();

 //Convert the response to a string
 int ch;
 while ((ch = responseStream.ReadByte()) != -1)
  ReturnStr = ReturnStr + Convert.ToChar(ch);

 //clean up
 responseStream.Close();
 response.Close();

 //split the string by new line
 string[] sep = {"\r\n"};
 string[] Files = ReturnStr.Split(sep, StringSplitOptions.RemoveEmptyEntries);
 return Files;
}

private void DownloadFile(string FileName)
{
 //Connect to the FTP
 FtpWebRequest request = WebRequest.Create("ftp://lcalhost/Completed/" + FileName) as FtpWebRequest;
 //Specify we're downloading a file
 request.Method = WebRequestMethods.Ftp.DownloadFile;
 request.Credentials = new NetworkCredential("SomeUser", "SomePassword");

 //initialize the Filestream we're using to create the downloaded file locally
 FileStream localfileStream = new FileStream(@"c:\Data\" + FileName, FileMode.Create, FileAccess.Write);

 //Get a reponse
 WebResponse response = request.GetResponse();
 Stream responseStream = response.GetResponseStream();

 //create the file
 byte[] buffer = new byte[1024];
 int bytesRead = responseStream.Read(buffer, 0, 1024);
 while (bytesRead != 0)
 {
  localfileStream.Write(buffer, 0, bytesRead);
  bytesRead = responseStream.Read(buffer, 0, 1024);
 }

 //clean up
 localfileStream.Close();
 response.Close();
 responseStream.Close();

}

Happy Programming!

Comments (32) -

Thanks!!!

This was extremely helpful :0)
I really appreciate the post!!!

Reply

Loads of thanks for the posting. It's really very helpful!!

Reply

great code thanks man

Reply

What all wildcards are supported by .Net?

Reply

hey mister!
What a shit! I did it with two lines not 30.
learn how to programm first, Bastardo.

Reply

Hi,
Just wanted to know, what will happen if the directory (which is viewed by ListDirectory) contains sub directories. And I want to download files located in that sub-directory too.
How can I know that which items listed by "ListDirectory" method are files and which are sub-directories?
Please acknowledge.

Reply

Andrew,
   There is no need to be rude. If you did it in 2 lines please share your code with us so we can see how you did it and learn from you.

Reply

Aman Sura,
   This only gets the files from a directory. If you would like to get all the files within subdirectories then you will need to look into recursion.

Reply

Hi,
Thanks a lot for quick reply.
I agree, recursion is the only solution here. But FtpResponseStream gives us a raw string as output. SO how do we make up, whicj item in the list if a file and which is a sub-directory, so that we can start recursion to fetch contents of a sub-directory?
I did it with string parsing. i.e. The directories contain a <DIR> but files do not. So i differentiated it from there.
Also, i could not find any other solution. Do you know of some other elegant solution?
Thanks again Smile

And people like Andrew should better keep their foul mouths shut..

Reply

Aman Sura,
     The only other way i can suggest of doing this is by checking if there is an extension. If it has an extension it is a file, otherwise it's a directory. In a sense is the same concept as yours, but yours is a more exact approach since a file could be extensionless.

Reply

Thanks Jonas for giving me your time.
Thanks a lot once again.

I think till the time .NET gives us some API to get list of files and directories, we have to deal with things this way only.

Reply

Hi,

Code is looking good and works fine. I have a problem: I want to download two types of files everytime. I want to download all files, which are of type .abc and .xyz. This a real requirement and urgent.

Thanks and regards,
rcm

Reply

Waseem aL Mustafa
Waseem aL Mustafa

Thanks very much for this helpful example

Reply

Thank you very much for posting this! This was extremely helpful.  I'd like to post a link to this on my blog, if I may.

Reply

@Jennifer - Go ahead, you may link to this post!

Reply

I bet you didn't think that after over 2 years this post would still be helping people. Thanks a lot for the post.

Reply

This was great - just what I needed.  Thanks!

Reply

Syed Hashmi
Syed Hashmi

Thanks. It really helped me. Please find C# code below.

        public void DownloadFiles(string WildCard)
        {
            //WildCard = "*Parts.csv"
            string[] Files = GetFiles(WildCard);
            foreach (string file in Files)
            {
                DownloadFile(file);
            }
        }

        private string[] GetFiles(string WildCard)
        {
         string ReturnStr = "";
         //Connect to the FTP
         FtpWebRequest request = (FtpWebRequest)WebRequest.Create("ftp://server/folder/" + WildCard);
         //Specify we're Listing a directory
         request.Method = WebRequestMethods.Ftp.ListDirectory;
         request.Credentials = new NetworkCredential("UserId", "Password");

         StringWriter sw = new StringWriter();
         //Get a reponse
         WebResponse response = request.GetResponse();
         Stream responseStream = response.GetResponseStream();

         //Convert the response to a string
         int ch;
         while ((ch = responseStream.ReadByte()) != -1)
          ReturnStr = ReturnStr + Convert.ToChar(ch);

         //clean up
         responseStream.Close();
         response.Close();

         //split the string by new line
         string[] sep = {"\r\n"};
         string[] Files = ReturnStr.Split(sep, StringSplitOptions.RemoveEmptyEntries);
         return Files;
        }

        private void DownloadFile(string FileName)
        {
            //Connect to the FTP
            FtpWebRequest request = WebRequest.Create("ftp://server/folder/" + FileName) as FtpWebRequest;
            //Specify we're downloading a file
            request.Method = WebRequestMethods.Ftp.DownloadFile;
            request.Credentials = new NetworkCredential("UserId", "Password");

            //initialize the Filestream we're using to create the downloaded file locally
            FileStream localfileStream = new FileStream(@"c:\temp\" + FileName, FileMode.Create, FileAccess.Write);

            //Get a reponse
            WebResponse response = request.GetResponse();
            Stream responseStream = response.GetResponseStream();

            //create the file
            byte[] buffer = new byte[1024];
            int bytesRead = responseStream.Read(buffer, 0, 1024);
            while (bytesRead != 0)
            {
                localfileStream.Write(buffer, 0, bytesRead);
                bytesRead = responseStream.Read(buffer, 0, 1024);
            }

            //clean up
            localfileStream.Close();
            response.Close();
            responseStream.Close();

        }

Reply

Great article this is just what I needed and thanks for your posting.

Reply

Shailendra
Shailendra

I am getting below error when i use ocl-fng.ilx.net ftp site on DownloadFile() method. Please advice.

Unhandled Exception: System.ArgumentException: Illegal characters in path.
   at System.IO.Path.CheckInvalidPathChars(String path)
   at System.IO.Path.Combine(String path1, String path2)
   at ftpdownload.DownloadFile(String FileName)
   at ftpdownload.DownloadFiles(String WildCard)
   at ftpdownload.Main()

Reply

if we have read only account of ftp site can we download files? if yes then please let me know the code.

Reply

Do I have to log into the remote site 100 separate times, to get 100 files?

Why not just log in once... and keep downloading until you have them all?

Reply

@Janice, you're not login in 100 times, you're making 100 requests with the credentials on each request. Only way to do it. If you find a better solution, let us know by posting a comment.

Reply

Here is a VB .net conversion:

Private Shared Function InlineAssignHelper(Of T)(ByRef target As T, ByVal value As T) As T
        target = value
        Return value
    End Function

    Public Sub DownloadFiles(ByVal WildCard As String)
        'WildCard = "*Parts.csv"
        Dim Files As String() = GetFiles(WildCard)
        For Each file As String In Files
            DownloadFile(file)
        Next
    End Sub

    Private Function GetFiles(ByVal WildCard As String) As String()
        Dim ReturnStr As String = ""
        'Connect to the FTP
        Dim request As FtpWebRequest = DirectCast(WebRequest.Create("ftp://Userid@server//directory/" & WildCard), FtpWebRequest)
        'Specify we're Listing a directory
        request.Method = WebRequestMethods.Ftp.ListDirectory
        request.Credentials = New NetworkCredential("UserId", "Password")

        Dim sw As New StringWriter()
        'Get a reponse
        Dim response As WebResponse = request.GetResponse()
        Dim responseStream As Stream = response.GetResponseStream()

        'Convert the response to a string
        Dim ch As Integer
        While (InlineAssignHelper(ch, responseStream.ReadByte())) <> -1
            ReturnStr = ReturnStr & Convert.ToChar(ch)
        End While

        'clean up
        responseStream.Close()
        response.Close()

        'split the string by new line
        Dim sep As String() = {vbCr & vbLf}
        Dim Files As String() = ReturnStr.Split(sep, StringSplitOptions.RemoveEmptyEntries)
        Return Files
    End Function

    Private Sub DownloadFile(ByVal FileName As String)
        'Connect to the FTP
        Dim request As FtpWebRequest = TryCast(WebRequest.Create("ftp://Userid@server//directory/" & FileName), FtpWebRequest)
        'Specify we're downloading a file
        request.Method = WebRequestMethods.Ftp.DownloadFile
        request.Credentials = New NetworkCredential("UserId", "Password")

        'initialize the Filestream we're using to create the downloaded file locally
        Dim localfileStream As New FileStream("c:\temp\" & FileName, FileMode.Create, FileAccess.Write)

        'Get a reponse
        Dim response As WebResponse = request.GetResponse()
        Dim responseStream As Stream = response.GetResponseStream()

        'create the file
        Dim buffer As Byte() = New Byte(1023) {}
        Dim bytesRead As Integer = responseStream.Read(buffer, 0, 1024)
        While bytesRead <> 0
            localfileStream.Write(buffer, 0, bytesRead)
            bytesRead = responseStream.Read(buffer, 0, 1024)
        End While

        'clean up
        localfileStream.Close()
        response.Close()
        responseStream.Close()

    End Sub

Reply

Wow Your are Great.. Thank you very much..

Reply

I got problem: Unable to connect to remote server, please help thanks.

Reply

Hi , when I host my website, downloadfile function cannot connect to remote server.

Reply

how to run this VB script .. i want to download file from mainframe ...reply mee soon
thakq very much

Reply

wow. great work, saved my time

Reply

Thank you for the post! You are a life saver

Reply

Add comment

biuquote
Loading