Announcement

Collapse
No announcement yet.

Sending very large files between networks

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • Sending very large files between networks

    Hi,
    I'm a software developer.
    I have an application which needs to sends very large files
    between networks all over the world.
    Once an hour (approximately) the application must send 2GB file
    to a remote server on another network in the other side of the world.

    From both IT perspective and software development perspective – what is
    the most reliable way to perform such a task?
    Is there some well-known product for this? Is ftp (client in code) enough reliable
    to send such large files?
    Should I use lower-level protocol for this?

    Our code is using MS technologies (.net).

    Thanks.

  • #2
    Re: Sending very large files between networks

    How sensitive is the information in the files?
    1 1 was a racehorse.
    2 2 was 1 2.
    1 1 1 1 race 1 day,
    2 2 1 1 2

    Comment


    • #3
      Re: Sending very large files between networks

      FTP is the way to go in general. However there are many people who are wary of FTP for long term use. FTP can be finicky at times so you really have to build in checks to make sure that all data was send across correctly, outside of FTP. (checksum, size, date, etc...)

      if you are brave you can check out rsync for windows here, although I have never seen it embedded within a windows app. You really would be on you own if you go there.

      Finally I must admit that I am by no means a programmer...
      "...if I turn out to be particularly clear, you've probably misunderstood what I've said” - Alan Greenspan

      Comment


      • #4
        Re: Sending very large files between networks

        rvalstar just reminded me of SFTP. so that is an option as well for securing the data. see here
        "...if I turn out to be particularly clear, you've probably misunderstood what I've said” - Alan Greenspan

        Comment


        • #5
          Re: Sending very large files between networks

          You also want to make sure that if you have any network hiccups, whatever method you use can / will automatically restart where it left without having to start from the beginning of the file. Similar to a download manager (pull) but in a push configuration. Don't know which FTP / SFTP clients offer that off hand so do your research.

          A long time ago with a really bad network connection and no ability to implement a traditional restartable method, I would run a script that would break the image into 1 MB files on the source then do a move from the source file share to the destination over a VPN. If the script encountered an error, it would sleep a bit then restart. When all the files were gone from the source, another script would merge them all back together on the destination.

          Point being, "where there is a will there is a way".
          Cheers,

          Rick

          ** Remember to give credit where credit is due and leave reputation points sigpic where appropriate **

          © 2006-2099 R Valstar. This post is offered "as is" for discussion purposes only with no express or implied warranty of any kind including, but not limited to, correctness or fitness for use. Nothing herein shall be construed as advice. Attempting any activity based on information in this post is done at your own risk.

          Comment


          • #6
            Re: Sending very large files between networks

            Originally posted by johnyGo View Post
            Once an hour (approximately) the application must send 2GB file
            to a remote server on another network in the other side of the world.

            F.

            First very important question is do you have end to end VPN connection between your location and the other location which is somewhere abroad ? If there is dedicated VPN connection you don't need to worry about encryption. In 1 hour you'll be able to transfer 2 GB data depending upon traffice between your VPN .

            Sending this via FTP may be the good solution. But again it depends on the bandwidth and traffic. If firewall is at both ends it'll not be easy. Many facts to consider. Where you are located and the location is where? If the other location is somewhere in Asia, don't believe that you can transfer 2 GB data across in 1 hour.
            Break those files and calculate its "HASH" value so at other end user can verify wether file is ok or damaged before he use it in there environment.
            All in 1
            Solaris,Linux & Windows admin + networking.

            Comment


            • #7
              Re: Sending very large files between networks

              Thanks u all guys for you advices.
              generally, i would appreciate any IT advice regarding the following scenario:
              There are few branches of some company, each branch serve different aspect
              of production line, the data being sent is large binary files describing some
              product (for example, huge grahic vector file to describe some product ).

              What i have currently in mind is:
              VPN between the branches, FTP site in each, web services in each.
              Each branch will call other branch sftp site, send file, then call the
              web service to confirm the arrival (and to invoke some command).

              What do u think?

              Comment


              • #8
                Re: Sending very large files between networks

                Don't know if WebDAV would be of any use but it may be worth a look.
                http://technet2.microsoft.com/Window....mspx?mfr=true

                http://support.microsoft.com/kb/323470
                1 1 was a racehorse.
                2 2 was 1 2.
                1 1 1 1 race 1 day,
                2 2 1 1 2

                Comment


                • #9
                  Re: Sending very large files between networks

                  Originally posted by johnyGo View Post
                  VPN between the branches,...
                  ...Each branch will call other branch sftp site,...
                  Wouldn't this be a little superfluous? SFTP would encrypt the data and when it hits the VPN tunnel, it would then be encrypted again. Do you need that kind of security?
                  Regards,
                  Jeremy

                  Network Consultant/Engineer
                  Baltimore - Washington area and beyond
                  www.gma-cpa.com

                  Comment

                  Working...
                  X