• Sun Blog Comments Off on How FAST is your Storage?

    USENIX puts on a storage conference every year called FAST: File and Storage Technologies. Sun is a Platinum sponsor this year and is holding its semi-annual OpenSolaris Storage Summit on Monday. The summit is free and is being held at the same hotel (San Francisco Grand Hyatt) as the FAST conference. There is a wiki that you can edit to register for the summit, and use to find out more about the event.

    The FAST conference itself is a great place to find out the latest research from the academic side of the storage industry. I plan to attend the technical sessions. One presentation in particular I am interested in is on Thursday. Brandon Salmon is talking about a system he helped design that manages all the data on your devices (laptops, desktops, PDAs, Phones) and controls the redundancy and location of that data based on how much you value it (or fear it’s loss). I met Brandon at the annual Parallel Data Lab (PDL) retreat put on by CMU. Sun is also a sponsor of PDL.

    I will be involved in a couple of Birds of Feather session during FAST. On Wednesday night, you can find me at the Open Storage BOF, where we will discuss the use of open source software to create storage solutions. Thursday night I’ll be leading a BOF on Cloud Storage, a trend that seems to be getting some traction lately. There is some interest in creating standard APIs for this type of storage, while others feel that standards at this early stage might stifle innovation. Should be an interesting discussion. Hope to see you there.

    I'm going to FAST '09

    Update: original Sun Blog post: 2/20/09

  • Sun Blog Comments Off on Is SMI-S “Done”?

    The blog-o-sphere has been abuzz of late with postings claiming that is dead, or dying. The claim being that the standard API/protocol has not fully replaced the need for proprietary ones. As if any management standard ever did. Perhaps if we all stopped adding features to our products, the standard could finally catch up 😉 You really can’t compare management standards to to protocols such as TCP/IP and HTTP, where there is a strong bias *against* adding features (witness IPv6).

    All these issues are hot topics within the standards organizations such as the SNIA and DMTF as well. Eight years later, the scope of resources that are able to be managed via a single protocol and common model has grown dramatically. The set of tools, mostly open source, to create solutions in this space is very impressive. What puzzles me is why anyone would re-invent all this infrastructure from scratch?

    Mostly, the use of proprietary protocols and APIs is historic. They came first. SMI-S, like any other “check box” feature gets thought about later in the development. Because the vendors don’t see it as “strategic”, they only implement the minimum to pass the certification tests.

    Once this gets established in a product line, it’s difficult to switch things around, but there really is no excuse anymore to start a new product line with a proprietary API as the “primary” means of managing a resource. The standards can easily be extended to cover the whole functionality of any given product. By leveraging SNIA and DMTF profiles and protocols as the primary interface, the implementation becomes rich and robust because there is no other interface.

    Some vendors do this today, but you’d never know it. What needs to happen is to have this become a differentiator between products, and get customers to ask for vendors to provide the full functionality through the standard-based interface. We really don’t need to standardize any more features in order for this to take place. It’s more of a messaging problem than a technical one.

    SMI-S may never be “Done” as long as we keep innovating in the storage space, and other implementations drive a need for a common way to manage those features, but who cares? It’s certainly complete enough to be the primary interface into your next storage purchase, if you call the vendor on it.

    Update: original Sun Blog post: 13/5/08

  • Sun Blog Comments Off on A New Model for the Storage Industry

    A this year’s Storage Developer Conference, I gave a talk about a new model that the SNIA Technical Council has been working on. It’s called the Storage Industry Resource Domain Model and we are using it to align various Technical Work Group (TWG) charters and to provide a baseline for new terminology in this space. We expect the industry to use it to map their products onto the model and talk in industry standard terms about what their products do.

    Here is the video:

    You might also want to download the iPod version for offline viewing.

    The slides can be downloaded from here

    Update: original Sun Blog post: 10/6/08

  • Sun Blog Comments Off on Get Your Hands on XAM

    SNIA has recently blessed the 1.0 version of the eXtensible Access Method (XAM) standard. The specification will soon be sent to ANSI and ISO for national and international standardization, but you can get started on coding to XAM now.

    SNIA has also released the first working draft version of the XAM SDK. Download the source code from here (BSD License). This includes the source for a complete C XAM Library, a complete Java XAM Library and an early version of the Reference VIM that runs on any Java platform using a plain file system as the backing store.

    If you are new to XAM and want to learn more, there is a unique opportunity to get educated at the upcoming Storage Developers Conference in late September.

    SNIA_SDC_1.jpgThere will be a XAM Tutorial on Monday, XAM Hands on Lab for Developers (HOLD) sessions on Monday and Tuesday and more XAM related session talks on Tuesday as well. SDC goes all week until Thursday with even more content on other storage technologies you won’t want to miss.

    The XAM Initiative is planning to fund a limited number of scholarships to SDC for XAM Developers that otherwise would not be able to afford the registration fee (as much as $1395). If you are interested in the scholarship, you will need to fill out this online form to apply for the scholarship. We will be evaluating the entries based on XAM usage and need.

    Once you get your hands on the XAM SDK and have your hands on experience at SDC, be sure and sign up for the XAM Developers Group where you can ask questions of the experts and get help with your development. The experts and other developers there can give you a hand.

    Update: original Sun Blog post: 8/1/08

  • Sun Blog Comments Off on OpenSolaris Storage Summit


    The date for the OpenSolaris Storage Summit has been set. We will be hosting the event at the Santa Clara Hyatt Regency hotel on the 21st of September, 2008. This is right before this year’s Storage Developer Conference at which Sun is a Platinum sponsor.

    We already have a couple of keynote speakers lined up: Ben Rockwood and Mike Shapiro. It will take place all day Sunday, so if you are coming to SDC this year, come a day early and participate. We will have the first ever OpenSolaris Storage Community meeting (face to face). A great way to meet some folks that you may only know by their email addresses. Everybody has an opportunity to give a Lightning talk and/or put a poster together about what they are working on or doing with OpenSolaris storage.

    Registration is via a wiki page here: http://www.genunix.org/wiki/index.php/OpenSolaris_Developer_Summit_08. Just add your name and contact information into the Attendance List table and you are registered! Attendance is free and we hope many folks from the community will attend this year. Also, you should add yourself to the summit list, even if you are not sure you will be able to go, so you can keep up to date with the planned activities. We may create ways to participate even if you can’t make it to Santa Clara this year (if we get people asking for access). Once registered, scroll down to the bottom of the wiki and suggest some topics or volunteer to participate as a poster or lightning talk. Fill in some details about yourself (or someone else) at the bottom of the page.

    Check it out, get involved, and I hope to see you there! It will be a blast. More details later…

    Update: original Sun Blog post: 8/1/08

  • Sun Blog Comments Off on OpenSolaris FCoE Target Code Released

    Sun has released the source for the Fibre Channel over Ethernet (FCoE) target driver on OpenSolaris. Note that this is a complete software implementation of FCoE – no special HBA is required. It will operate with a simple Ethernet NIC as hardware (e1000g/bge). It will interoperate with the Linux initiator from the Open-FCoE project.

    Modern processors (with multiple cores and threads) have plenty of cycles to do software initiators and targets. Software initiators and targets for iSCSI are pretty much the norm (TCP offload never took off). In the case of FCoE, does it make economic sense to offload the Fibre Channel stack to an offload engine any more than the TCP/IP stack? But that’s what many of the existing Fibre Channel HBA vendors are hoping for with FCoE – to sell you an offload stack and charge more for the NIC.

    It’s too early to tell how this new protocol will shake out in the products that survive in the market, but you can get started now with this newly released code. Download it, play with it, do some performance measurements and get involved in the OpenSolaris community.

    Update: The OpenSolaris FCOE Initiator has now also been put back.

    Update: original Sun Blog post: 7/31/08

  • Sun Blog Comments Off on So you want to build a storage server…

    You have lots of company. Storage Servers have been around since the days when Sun first put NFS on their workstations. Administrators would load up one workstation with disks and share it with the others. Even today, a large percentage of our “servers” end up as storage servers when deployed.

    The acquisition rules for some companies actually favor building a storage server over the purchasing of a separate appliance. The server budget is typically separate from the storage budget for most organizations. Administration is more familiar as well. Administering a storage server is little different from managing any other server, save for the additional commands required to expose the internal storage to a separate host.

    So when building a storage server, you want to understand your requirements of course. File or Block protocols? Ethernet or Fibre Channel or even SAS to the hosts? How will you protect the data: snapshot, backup, remote replication? How much CPU power do you need? How much storage space to match that? All of these factor into which server you should buy or repurpose for this deployment and which OS you install. Thumper.jpg

    Sun does have servers that make a good choice for a storage server. Of course the classic example is the 4500 “Thumper” storage server. Customers have been buying these from us in droves, which is the best proof of the value of this notion. With the OpenSolaris storage stack they have been able to make them into NFS servers, CIFS server and even iSCSI arrays. Now Sun has released software that will take storage servers to the next level: Multi-protocol storage arrays.

    The new software, part of the latest build of OpenSolaris, is affectionately called COMSTAR and boy is it the STAR of the storage COMmunication you want to be able to do with a storage server. Fibre Channel, SAS, iSCSI, iSER, OSD, you-name-it, if it’s a storage communication protocol, COMSTAR handles it.

    Now it may seem silly to think about putting a bunch of SAS/SATA drives in a box with a CPU and then just re-exporting them as SAS targets to hosts. What value does this have? Of course it’s what you do in that storage server before you re-export them that make this a storage server and not just an enclosure. You want mirroring, disc scrubbing and automatic failover and repair – load up ZFS. You want remote replication – load up Availability Suite. You want to be able to diagnose bottlenecks and performance issues – load up DTrace. Then export the resulting highly available highly performant virtualized disks.

    I think our current stars: ZFS and DTrace, should make room for the new star. Let’s hear it for ZFS, DTrace and COMSTAR. Hooray!

    Update: original Sun Blog post: 6/18/08

  • Sun Blog Comments Off on XAM Java Binding Overview

    This article was originaly posted to the Royal Jelly blog: http://weblogs.java.net/blog/macsun/

    The eXtensible Access Method (XAM) is new kind of storage interface for storage devices that can handle a special type of data, called “Fixed Content“. Version 1.0 is soon to be released, but there is a working draft available of the Architecture Specification, the C Language Binding and the Java Language Binding. It is this last document that this post covers. A good introduction to XAM is the beginning chapters of the Architecture specification, and that document covers the normative behavior required for implementations of the interface.

    The primary Java Object that applications deal with to store and retrieve data is called an Xset. It is the object that is the container for both the data as well as the associated metadata. XSet metadata goes beyond the typical filesystem metadata, and adds fields for retention and other data management policies. Applications can also add their own metadata as well. This is a key requirement in the fixed content storage market. Any of the metadata in the XSet can be queried for later, in order to find the XSet.

    But before we dig into the XSet, let’s talk about the other primary objects of the XAM Interface. First is the notion of a system that will contain the XSets. We call this storage system an XSystem in the abstract. The XSystem object is also the factory for XSet objects (via create and open methods). Now any given host may have access to any number of XSystem storage devices, so we need a way to create XSystem objects for each system we need to access. We do this through an object called the XAMLibrary. The XAM Library manages all the connections to the storage devices and acts as a factory for XSystem objects. It does this through the connect() method:

    public XSystem connect(String XRI) 

    The XRI (XSystem Resource Identifier) string is a “handle” that is used to load the correct Vendor Interface Module (VIM) for the device and locate it, perhaps on the network. It has the following syntax:

    "snia-xam://" [vimname] xsystemname [params]

    The XSystem object that is returned from the connect() operation can now be used to talk to the storage device. The first thing we will need to do is to authenticate with the device, to prove that we are who we say we are. This is done through the authenticate() method:

    public byte[] authenticate(byte[] buffer) 

    See the Architecture Specification for a more detailed description of the steps required to authenticate to an XSystem.

    So we have the three primary objects of the interface: XAMLibrary, XSystem and XSet. All of these objects have properties which are accessed in the same way. These properties can be of several different types and can be created at runtime. Thus we need a way to do this in a type safe manner. The key to this capability is something called an XIterator:

    import java.util.Iterator;
    public interface XIterator extends Iterator 
        public void close() throws AuthenticationExpiredException, XAMException; 

    Many of the basic operations on the fields these primary objects contain are in common, so the XAM interface defines a superclass for the primary objects called FieldContainer:

    public interface FieldContainer 
     XIterator openFieldIterator(String prefix) 
    	throws AuthenticationExpiredException, InvalidFieldNameException, 
           ObjectInUseException, XSetAbandonException, 
           XSetCorruptException, XSystemAbandonException, 
           XSystemCorruptException, XAMException; 

    The XIterator returned from openFieldIterator() can contain all properties, or just a subset (those having a name starting with “prefix”). The FieldContainer Java interface also contains methods for: field interrogation; property creation, accessors, and mutators; field operations, and XStream operations. An XStream is an object that can contain the actual content associated with an XSet. XStreams have operations to read and write the content using various means both synchronously and asynchronously. The type of data an XStream holds can be any valid Mime type.

    This post covers most of the basic objects that are part of the XAM interfaces. Future posts will delve into specific use cases.

    Update: original Sun Blog post: 6/16/08

  • Sun Blog Comments Off on Preservation and Archiving

    It was an exciting week for Preservation and Archiving activities.

    First, I was introduced to several key academic researchers (and software developers) in the digital repository space who had come early to get the inside scoop on the latest ST 5800 (Honeycomb) developments. Dave Tarrant (University of Southampton), Ben O’Steen (University of Oxford), and Neil Jeffries, R&D Project Manager, Oxford University Libraries met with us in the Honeycomb development facility in San Francisco. I have discussed Open Archives before, but these folks are working on some great software that starts to fulfill that promise. They are also customers of the ST 5800 and are thrilled that we just Open Sourced the latest (1.1.1) product code and can contribute some of their ideas to that code base in the Honeycomb community.

    Which brings us to the event this last week that I attended. Sun sponsors a regular set of meetings of what is called the Preservation and Archiving Special Interest Group (PASIG). The latest event was held last week in San Francisco and boasted attendance of over 130 non-Sun attendees from across the industry. There were lots of great talks given, and I highlight a few below:

    Neil Jefferies talked about the Oxford Digital Asset Management System (slides). They use FEDORA-Commons in combination with the ST 5800 and other open source projects. They are ingesting the output of the Google Library Project into DAMS, something they call Mass Digitisation Ingest Components (MDICS).

    David Tarrant talked about his work with EPrints at the University of Southampton (slides). His emphasis is on supporting small science as well as big science projects. He also talked about some of the requirements for long term storage. He has combined the open source EPrints repository software with the ST 5800, utilizing the self managed storage to extends the value of the repository. EPrints has an extensible plug-in architecture that allows for any storage device or database to be plugged in. He also discussed the Preserv interoperabilty project that is working towards standards that separate the reository software from the storage controller module and from the actual physical storage.

    Ray Clarke gave a great talk (slides) on some of the best practices for long term data retention. He also gave a talk (slides) on some of the activities at the SNIA in this area including XAM, the Data Management Forum (DMF) and the new Long Term Retention Technical Work Group (LTR TWG). SNIA is working on a format for encapsulating data along with its metadata that will allow it to preserved in a storage system independant manner over its lifetime. Most of the attendees of the conference expect to retain and preserve their data essentially forever.

    There were lots of other great talks which you can find here. Many thanks to Art Pasquinelli (and many others) who did a bang up job as usual organizing the event. The next meeting is tentatively scheduled for November in Europe. If you want to join the Sun PASIG community, join here.

    Update: original Sun Blog post: 6/5/08

  • Sun Blog Comments Off on Don Deel: Newest SNIA TC member

    I first met Don in the early days of the Bluefin effort when we were both helping start what would eventually become the SMI-S standard. He had his own SAN management software startup called Prisa Networks with a pretty comprehensive Java based framework for management. At the time, I was trying to convince him to adopt Jiro, as I recall.

    Now Don has been appointed to fill the seat recently vacated by David Black on the SNIA Technical Council. Most recently Don and I were co-chairs for the Management Frameworks TWG, which then produced a reference architecture.

    I recently interviewed Don for the Technical Council Podcast series. You can subscribe (iTunes) or just download the .mp3.

    Welcome Don! I know you will make some valuable contributions in your new role.

    Update: original Sun Blog post: 5/18/08

« Previous Entries   Next Entries »

Recent Comments