Announcement

Collapse
No announcement yet.

Best methods for long term storage of data

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • Best methods for long term storage of data

    Hi all,

    I am looking for advice /suggestions really on the best long term data storage methods.

    Recently the company I work for has finally got around to reducing its huge archives of paper invoices and other accounts related documents. At the moment they are being scanned in (PDF format) and I estimate ultimately there will be over 1.5TB of digital data.

    I am really looking for the best way to store this data for a long time.

    I guess I should store on at least 2 different media formats in different geographical locations but which ones?

    Hard drive (RAID)? Blu-Ray Disc? Tape?

    Also what steps should I take to maintain data integrity

    How often should I test the data? How should I store it.

    I am looking at prospective timescale to keep the data of up to 30 years

    Thanks for your input

    Dave

  • #2
    Re: Best methods for long term storage of data

    Funnily enough, this is being discussed in the abstract at this very moment:
    http://forums.petri.com/showthread.php?t=46451

    IMHO hard drives are probably the best way to go, with multiple copies in separate locations (contact Iron Mountain if you want the nuclear bunker option). Keep at least 3 copies so you can check which copy is good if one is damaged - with two copies you can never be sure. Also calculate hash values (the sort of thing forensic software does) so you can verify the contents of the drive have not changed.

    Rather than a RAID array which is running, use removable drives and store them offline, so there will be no mechanical wear and tear. if you are really paranoid, consider buying additonal drives of exactly the same model so the electronics can be swapped out. Get metal containers that act as Faraday cages to prevent EMP!

    Testing: not too often but often enough you are happy it works -- say twice a year?
    Tom Jones
    MCT, MCSE (2000:Security & 2003), MCSA:Security & Messaging, MCDBA, MCDST, MCITP(EA, EMA, SA, EDA, ES, CS), MCTS, MCP, Sec+
    PhD, MSc, FIAP, MIITT
    IT Trainer / Consultant
    Ossian Ltd
    Scotland

    ** Remember to give credit where credit is due and leave reputation points where appropriate **

    Comment


    • #3
      Re: Best methods for long term storage of data

      Mmmm Must have missed that one in my clearly insufficient pre-post searches!

      Thanks a lot. My first though was hard drives but you have added a few other points I would have probably overlooked. As for the nuclear bunker option, I dont think we are quite ready for that yet!

      Thanks a lot as ever!

      Comment


      • #4
        Re: Best methods for long term storage of data

        No problem
        Of course, I am making an assumption that USB or Firewire will be around in 30 years -- better add another point to make a regular review and migrate to new technologies if needed!
        Tom Jones
        MCT, MCSE (2000:Security & 2003), MCSA:Security & Messaging, MCDBA, MCDST, MCITP(EA, EMA, SA, EDA, ES, CS), MCTS, MCP, Sec+
        PhD, MSc, FIAP, MIITT
        IT Trainer / Consultant
        Ossian Ltd
        Scotland

        ** Remember to give credit where credit is due and leave reputation points where appropriate **

        Comment


        • #5
          Re: Best methods for long term storage of data

          I had considered that. I guess that would most likely be part of the bi-yearly test!

          Comment


          • #6
            Re: Best methods for long term storage of data

            1.5TB of data is not a problem for a HDD since there are 2TB drives and 4TB drives are supposed to be out late this year. If you can store the data on one HDD instead of a RAID then it reduces the possibilty of data loss in the case of a RAID Controller failure. 2 or individual HDDs would do the trick. Make sure you get HDDs that have way different Serial Numbers. I have seen it where you can have problems when a batch of drives have a problem and you find the S/Ns are in a close sequence. Remember WD recalling 400,000 drives a few years ago when they were supplied with a faulty chip?

            I would avoid putting a backup on tape because the tape drive will fail when you need to use it in 5 or 6 years time and they won't make that unit anymore so you can't read the tape. Tapes are also susceptable to flaking and magnetic interference. (An accountant who was responsible for taking the off-site tape home was terminated when the on-site and near-site backups failed and then the off-site ones did too. He used to put the tapes on the seat of his Saab and then turn the seat heater on. MMmmm, nice electromagnetic field. Personally the tech should have been shot, drowned and then fired from a canon for not running any test Restores on any of the backups)

            Depending on the length of time you want to store this data then Blu-ray may be an option if you can be sure the disks will maintain their integrity that long. Just remember that CDs were supposed to last forever but the first ones started rotting away after 15 years. I just hope my Kodak Gold ones last a bit longer than that.
            1 1 was a racehorse.
            2 2 was 1 2.
            1 1 1 1 race 1 day,
            2 2 1 1 2

            Comment


            • #7
              Re: Best methods for long term storage of data

              Of course, there is always the option of making a hard copy on high quality paper and storing that
              Tom Jones
              MCT, MCSE (2000:Security & 2003), MCSA:Security & Messaging, MCDBA, MCDST, MCITP(EA, EMA, SA, EDA, ES, CS), MCTS, MCP, Sec+
              PhD, MSc, FIAP, MIITT
              IT Trainer / Consultant
              Ossian Ltd
              Scotland

              ** Remember to give credit where credit is due and leave reputation points where appropriate **

              Comment


              • #8
                Re: Best methods for long term storage of data

                Yes, thanks for the tip about the hard drive serial numbers. I can just imagine my luck getting three from the same faulty batch! ...disaster!

                Comment


                • #9
                  Re: Best methods for long term storage of data

                  @Ossian

                  yes, that has been done ..and now fills a significant portion of a warehouse! Hence the scanning!

                  I think the buzz words 'Paperless office' have been mentioned so many times in the last few weeks they have become permenantly lodged in my brain!

                  Comment


                  • #10
                    Re: Best methods for long term storage of data

                    It was a joke, honest....
                    Although there is some truth... books and paper are readable easily after hundreds of years -- can any digital medium claim that? (5 1/4 inch floppy, anyone?)

                    Note, that "Paperless Office" is as much an oxymoron as "Microsoft Works"
                    Tom Jones
                    MCT, MCSE (2000:Security & 2003), MCSA:Security & Messaging, MCDBA, MCDST, MCITP(EA, EMA, SA, EDA, ES, CS), MCTS, MCP, Sec+
                    PhD, MSc, FIAP, MIITT
                    IT Trainer / Consultant
                    Ossian Ltd
                    Scotland

                    ** Remember to give credit where credit is due and leave reputation points where appropriate **

                    Comment


                    • #11
                      Re: Best methods for long term storage of data

                      Yes indeed! Microsoft Works ....

                      Comment


                      • #12
                        Re: Best methods for long term storage of data

                        You also need to consider the format for the PDF's. We have to submit PDF versions of the results of projects we undertake. Because our work destroys the resource we are studying (we are archaeologists) the only record available is the one we create. Recently, the national body that accepts these PDF versions has specified that they prefer PDF/A files, or PDF-Archival standard.

                        http://www.oasis.ac.uk/faq.cfm

                        Check out the Questions from Contractors section

                        I'm sure there are far more comprehensive resources available about this.
                        A recent poll suggests that 6 out of 7 dwarfs are not happy

                        Comment


                        • #13
                          Re: Best methods for long term storage of data

                          The point of USB/Firewire's availability in 30 years time is a good one - I'm sure there are some of us on this forum who remember the good old days of MFM, and miss it as much as I do *.

                          What are the reasons for moving from paper to digital - is it purely the amount of space that's being used, or is it a push from management to adopt the latest cool innovation? Realistically, the paperless office will come along shortly after the paperless toilet

                          Originally posted by Ossian View Post
                          Note, that "Paperless Office" is as much an oxymoron as "Microsoft Works"
                          Britain's Got Talent...



                          * about as much as I miss visits to the proctologist.
                          Gareth Howells

                          BSc (Hons), MBCS, MCP, MCDST, ICCE

                          Any advice is given in good faith and without warranty.

                          Please give reputation points if somebody has helped you.

                          "For by now I could have stretched out my hand and struck you and your people with a plague that would have wiped you off the Earth." (Exodus 9:15) - I could kill you with my thumb.

                          "Everything that lives and moves will be food for you." (Genesis 9:3) - For every animal you don't eat, I'm going to eat three.

                          Comment


                          • #14
                            Re: Best methods for long term storage of data

                            Please, no obscenities like that here!
                            Tom Jones
                            MCT, MCSE (2000:Security & 2003), MCSA:Security & Messaging, MCDBA, MCDST, MCITP(EA, EMA, SA, EDA, ES, CS), MCTS, MCP, Sec+
                            PhD, MSc, FIAP, MIITT
                            IT Trainer / Consultant
                            Ossian Ltd
                            Scotland

                            ** Remember to give credit where credit is due and leave reputation points where appropriate **

                            Comment


                            • #15
                              Re: Best methods for long term storage of data

                              We use Commvault to archive our older data to tape which is then stored off site. Commvault then archives the data to our SAN and that data is replicated using EMC RecoverPoint devices to our SAN in our DR site.

                              I would never trust optical media as I have seen many name brand DVDs fail. You might also want to consider removeable SSD drives considering they have no moving parts!
                              |
                              +-- JDMils
                              |
                              +-- Regional Systems Engineer, DotNet programmer & Jack of all trades
                              |

                              Comment

                              Working...
                              X