1. Introduce

XData is a serialize product developed by xcore.top, very like json , but is much faster and small than it. What is most useful is that XData supplies a protocol design platform on which you can easily define your interface and generate the code automotically,then you can transport the object in binary data from one platform to another. Util now,we support Java,Objective-C and Javascript, so XData is very suitable to apply in the mobile development,and we are planning to support more platforms

2. What win

XData win other technologies in many dimensions: (we called 4S:Swift,Small,Simple,Security)

2.1 Swift

Speed is the one of the most important performance index of serialize technologies. Although the CPU and network developed very fast, we still concern with this first. and detailed data compared with json and protocol buffer will be disposed later, here,we just tell you why

  • Class structure is defined from the viewproint of serialize,not from the viewpoint of general program. so it is to say, born to be swift
  • Write object directly at once, no need to analysis the structure of object to determine the buffer size
  • Parse object finished in once a scan of source array,no need any duplicated read and slice on the source bytes
  • No reflection is needed in both of serialize and deserialize(This is the very reason why some technology is very slow at the first time of serialize and deserialize)
  • Use linked buffer unit when buffer need to be expanded,thus,cause less GC(This is the very reason why some technology gets slower as data grows up)
  • 2.2 Small

    The footprint of XData is very small and the contract class file is small as well

  • xdata.jar is only about 40Kb, as comparison, fastjson.jar is about 1.5MB
  • xdata class file is very like common JavaBean ,as comparison, files generated by protocol buffer is almost thousands of lines and hard to read
  • 2.3 Simple

    Have you ever seen a powerful serialize technoloy wrote in about 2000 lines? It must be simple

  • To understand a framework of 2000 lines would not be a difficult work
  • No use any complex design pattern and third part framework
  • No configuration need,Just open to use
  • Integration is very easy, you can use any exist framework,even that ,you don't need other framework, servlet , socket, or urlconnection is enough. And xcore.top is only a servlet
  • 2.4 Security

  • Binary based,No attribute is exposed, it is hard to read the data for network attacker
  • No reflection , All class , all fields can be proguard. the business meaning is hard to guess in static and dynamic debug
  • 3. How to use

    3.1 Register and setup project

    After your registeration, you can setup a project, then you can define an interface, for example, you want to create an login interface for your own project, to be simplified, you define your login data contains two fields, namely,mobile and password, they are both String

    3.2 Define your interface

    The platform helps you to define the classes. you just need input the information of your interface, assume you need a class like

                        package com.example;
                        public class LoginModel {
                            public String mobile;
                            public String password;
                        }
                    
    you just need to input the package name, class name,field name and select a type for the field , then use can generate the code for LoginModel. However,the LoginModel class you get will not be a simple POJO or Javabean, it is a wrapper class with speial structure ,and it is the speial structure that makes the serilize process fast and small. And we will discuss the structure later

    3.3 Generate code and download

    Download the xdata-v1.0.jar and add to libs of your projects. download your protocol source code generated by the platform. Then you will get a zip file, unzip to your project source directory(you should make sure the the source code located in the right place), then you can use the class right now. So assume that ,you can find the LoginModel reference now

    3.4 Serialize and Deserialize

    Serialize is very simple like below:

                        LoginModelWrapper loginModel = new LoginModelWrapper();
                        loginModel.setMobile("13566767891");
                        loginModel.setPassword("123456");
                        XDataWriter writer = new XDataWriter();
                        byte[] result = writer.writeData(loginModel);
                    
    This is the most common use case of serialize, then you can transfer the byte[] result to anywhere you like

    Deserialize is simple as well:

                        byte[] bytes = IO.readbytes(inputStream);// read the prior LoginModel bytes transfered from other platform
                        XDataParser parser = new XDataParser();
                        XData data = parser.parse(bytes);
                        LoginModelWrapper loginModel = new LoginModelWrapper(data);
                    
    This is the most common use case of deserialize(be careful, there is realy no any reflection)

    after you get the loginModel received, you can call methods to get data as you need:

                        System.out.println(loginModel.getMobile()); // print '13566767891'
                        System.out.println(loginModel.getPassword());//print '123456'
                    
    You see, the remote object(only data structure)is transfered to local successfully.

    All the classed generated by the platform can be serialized by XDataWriter and deserialized by XDataParser.However, you can hand write the classes as well, but we don't suggest to do so, because it is a little difficult and complex due to the binary operation (But, I think it is much easier than the proto file of protocol buffer).Anyway, if you realy want to have a try, all of xdata is open to you, and the the original source code of xcore.top are all hand write too, because there was not such platform for me

    4. Principle

    I believe that most of you are more curious of how can xdata be so simple than how powerful it is. Let's go on

    4.1 Type definition

    All type of carry some datas, for example ,int or Integer can carry a number ,String carry a sequence of chars. so the literal 'int','Integer' and 'String' are types to program language and they are almost exists in all languages. In deed, they are allocated different size of memory. and diffrent language may have diffirent access mechniasms. this cause that an object in one platform can't be access in another one. so there cames the serialization, convert all data in different platform to general bytes, then de data can be transfered to other platforms.so, if datatype definition (mostly we call it Class)is oriented to language, then we have to use the given language to copy then to byte array.However, if we consider from the other side, Class definition is oriented to serialize, then, Objects are no need to be serilized-they are already equal to raw byte array. That's what xdata does

    There is no syntax representation of int, long and so on in XData, as well as Class for the complex datatype. we use bit to explain the datatype of certain data. and each object has a field 'type' ,we have defined the following types:(Types)

                        /**1bit 整形,对应java boolean*/
                        int TYPE_BOOLEAN         = 0x00000100 ;
                        /**8bit 整形,对应java byte*/
                        int TYPE_BYTE_i_1        = 0x00000200 ;
                        /**16bit 整形,对应java short*/
                        int TYPE_BYTE_i_2        = 0x00000300;
                        /**32bit 整形,对应java integer*/
                        int TYPE_BYTE_i_4        = 0x00000400;
                        /**64bit 整形,对应java long*/
                        int TYPE_BYTE_i_8        = 0x00000500;
                        /**32bit 浮点,对应java float*/
                        int TYPE_BYTE_f_4        = 0x00000600;
                        /**64bit 浮点,对应java double*/
                        int TYPE_BYTE_f_8        = 0x00000700;
                        /**字符串,对应java String*/
                        int TYPE_STRING          = 0x00000800;
                        /**二进制数据,对应java byte[]*/
                        int TYPE_BLOB            = 0x00000900;
                        /**日期*/
                        int TYPE_DATE       = 0x00000A00;
                        /**预留*/
                        int TYPE_UNKNOWN_1       = 0x00000B00;
                        /**预留*/
                        int TYPE_UNKNOWN_2       = 0x00000C00;
                        /**预留*/
                        int TYPE_UNKNOWN_3       = 0x00000D00;
                        /**预留*/
                        int TYPE_UNKNOWN_4       = 0x00000E00;
                        /**预留*/
                        int TYPE_UNKNOWN_5       = 0x00000F00; 
                        

    Like protocol,we use 8bit to indentify the field index of a class. what XData do further is that, all Types of XData are indexed as well. So each class have a TYPE_INDEX when declared. and the TYPE_INDEX allocation rule is below:

  • Object type is same as predefined type, just the value is different,so use custome type is same as predefined type
  • Object type begin from 0x00001FFF ,which is defined in Types,namely the lower 13bits have special meaning,the remaining 19bits can allocated to represents a Object type
  •                             int TYPE_OBJECT_START    = 0x00001FFF;
                        
  • We are also reserved some types that will be used to by hand write, so the platform will allocate the type from: (and if you want to hand write the code, you should allocate from)
  •                             int TYPE_PROJECT_START  = ((TYPE_OBJECT_START >> 13) + 300) << 13;
                        

    so if you want to hand write, you' better write all TYPE_INDEX allocated in one file to avoid mistake,like:

                                public interface CType {
                                    int TYPE_CLASS_NAME_1          = ((XType.TYPE_PROJECT_START >> 13) + 1) << 13;
                                    int TYPE_CLASS_NAME_2          = ((XType.TYPE_PROJECT_START >> 13) + 2) << 13;
                                    int TYPE_CLASS_NAME_3          = ((XType.TYPE_PROJECT_START >> 13) + 3) << 13;
                                    int TYPE_CLASS_NAME_4          = ((XType.TYPE_PROJECT_START >> 13) + 4) << 13;
                                    int TYPE_CLASS_NAME_5          = ((XType.TYPE_PROJECT_START >> 13) + 5) << 13;
                                    int TYPE_CLASS_NAME_6          = ((XType.TYPE_PROJECT_START >> 13) + 6) << 13;
                                    int TYPE_CLASS_NAME_7          = ((XType.TYPE_PROJECT_START >> 13) + 7) << 13;
                                    int TYPE_CLASS_NAME_8          = ((XType.TYPE_PROJECT_START >> 13) + 8) << 13;
                                    int TYPE_CLASS_NAME_9          = ((XType.TYPE_PROJECT_START >> 13) + 9) << 13;
                                }                            
                        

    4.2 Field definition

    Like protocol buffer, XData class allocated 8bit to store the index of field. so a object can have 256 field in max , Not like protocol buffer, you do not need care about the index,the platform allocate the index automatically, and type of the field is OR to field index, and other mask OR to the final index of field

    for example, if you defined a integer field of certain custome type, if it is the 2rd field of the custom type, it's index is:

     
                             TYPE_BYTE_i_4 | 2
                             

    if this field is a list of integer, it's index is:

     
                             TYPE_BYTE_i_4 |TYPE_MASK_LIST | 2 
                        

    This is simple of how the custom type definition looks like:

                        public interface FieldInfo {
                            int TYPE_INDEX = CType.TYPE_FIELD_INFO;
                            int TYPE        = 1 | XType.TYPE_BYTE_i_4;
                            int IS_LIST     = 2 | XType.TYPE_BYTE_i_1;
                            int FILED_INDEX = 3 | XType.TYPE_BYTE_i_2;
                        }
                

    This defined a custom type, name is FieldInfo, it's TYPE_INDEX is CType.TYPE_FIELD_INFO,this type has 3 fields

  • TYPE is a integer with 4byte width
  • IS_LISt is a integer with 1byte width
  • FILED_INDEX is a integer with 2byte width
  • This inerface is a DSL of certain type of XData, usually, you will not like to use it, Howerer, it did can work like below code:

                    XData xdata = new XData(FieldInfo.TYPE_INDEX);
                    xdata.set(FieldInfo.TYPE,231);
                    xdata.set(FieldInfo.IS_LIST,1);
                    xdata.set(FILED_INDEX,5);
                    XDataWriter writer = new XDataWriter();
                    byte[] bytes = writer.writeData(xdata);
    
                    XDataParser parser = new XDataParser();
                    XData xdataRead = parser.parse(bytes);
                    System.out.print(xdataRead.getInteger(FieldInfo.TYPE));
                    System.out.print(xdataRead.getByte(FieldInfo.IS_LIST))
                    System.out.print(xdataRead.getShort(FieldInfo.FILED_INDEX))
                

    Someone likes this style. but someone thinks this style is to hard to use, so there comes the wraper to help, that makes it as usual:

                    FieldInfoWrapper wrapper = new FieldInfoWrapper();
                    wraper.setType(231);
                    wraper.setIsList(1);
                    wraper.setFieldIndex(5);
                    
                    XDataWriter writer = new XDataWriter();
                    byte[] bytes = writer.writeObject(wraper);
                    XDataParser parser = new XDataParser();
                    XData xdataRead = parser.parse(bytes);
                    FieldInfoWrapper readIn = new FieldInfoWrapper(xdataRead);
                    int type = readIn.getType());
                    byte isList = readIn.getIsList()))
                    short fieldIndex = readIn.getFieldIndex())
                

    This may satisfy the flavor of all kinds of engineer! But xcore.top uses the former because ,you know, there was no such platform for me!

    4.3 Serialize and deserilize

    This section seems to be important and complex, however, it is not, or, I think XData would not be so simple. It doesnot use any kinds of astraction or use any complex algorithm. By the way,In my opinion,a better model design is much useful than pattern and algorithm

    Let us remember how many serializers in fastjson and what is varint in protocol buffer , and how much that help performance and understanding? Now, see my code:

  • XDataWriter: 253 lines in total
  • XDataParser: 238 lines in total
  • and the source code is easy to obtain here.Just see see. Anyway, if you think it is difficult to understand, it deoesn't master,just use it!

    4.4 Types support

    Util now, we support the most common data types list in the following table

    typewidthdescription
    TYPE_BOOLEAN1bytesigned byte in java
    TYPE_BYTE_i_11bytesigned byte in java
    TYPE_BYTE_i_22bytesigned short in java
    TYPE_BYTE_i_44bytesigned int in java
    TYPE_BYTE_i_88bytesigned long in javascript
    TYPE_BYTE_f_44bytefloat in java
    TYPE_BYTE_f_88bytefloat in java
    StringdynamicString
    Blobdynamicbyte[]
    Date8byteDate
    XData Objectdynamiccompose of single or list of above
    ListdynamicList of above
  • Notice: Not like json, there is no XDataArray. because JsonArray make the user compfused- Is the parse result is a JsonObject or a JsonArray? you have to read the text then choose to write code. so there is no XDataArray. and Everything is XData
  • TYPE_BYTE_i_8 (long) respect the definition of javascript,absolutely is a double scope, max value is 53bit
  • Dynamic data' length is 4byte length, wrote before the real bytes
  • And, There are no Set and Map
  • We are not planning to support other literal types,yet