Bson to Document

[dariusc93]

What would be the best way to convert a bson to a document that is suitable for inserting into the database?

[zonyitoo]

Hmm, maybe we could define a trait, which could be used to convert a type to BSON serializable type?

[lord]

👍 ditto, was curious about this. It seems you can serialize to BSON with serde, but I can't figure out how to serialize a rust object into a document that can actually be inserted into the database.

[dariusc93]

It is not to hard to do it now @lord. I had to create a small wrapper to make it easy to do so.

[lord]

@dariusc93 mind sharing? I'm new to Rust, and can't quite seem to figure it out.

[dariusc93]

@lord here is what I use with rustc_serialize (you could change it so it would work with Serde).

Note: The variable DATABASE is a lazy static that holds the database connection so you may want to change it to init your database connection, and the Second 'coll_name' is there for testing.

pub trait Collections: Sized + Send + Sync + ToJson + Encodable + Decodable {

    type Type: Encodable + Decodable;

    fn collection_name(&self) -> &str;

    fn coll_name() -> &'static str;

    fn primary(&self) -> Document;

    fn insert(&self) -> Option<Self::Type> {
        let coll = DATABASE.collection(self.collection_name());

        let jsn = self.to_json();

        let jobj = match jsn.as_object() {
            Some(j) => j,
            None => return None
        };

        let data = {
            let mut doc = Document::new();
            for (key, value) in jobj.clone() {
                doc.insert(key, Bson::from_json(&value));
            }
            doc
        };


        match coll.insert_one(data.clone(), None) {
            Ok(_) => {
                let tmp: Self::Type = match json::decode(&jsn.to_string()){
                    Ok(t) => t,
                    Err(_) => return None
                };
                Some(tmp)
            },
            Err(_) => None
        }
    }

    fn find_all(document: Option<Document>) -> Option<Vec<Self::Type>> {
        let coll = DATABASE.collection(Self::coll_name());
        let mut data = match coll.find(document, None) {
            Ok(e) => e,
            Err(_) => return None
        };
        let mut list: Vec<Self::Type> = Vec::new();
        loop {
            let item = data.next();
            match item {
                Some(Ok(doc)) => {
                    let temp: Self::Type = match json::decode(&Bson::Document(doc).to_json().to_string()){
                        Ok(a) => a,
                        Err(_) => return None
                    };
                    list.push(temp);
                },
                Some(Err(_)) => return None,
                None => break
            }
        }
        Some(list)
    }

    fn find_one(document: Option<Document>) -> Option<Self::Type> {
        let coll = DATABASE.collection(Self::coll_name());
        let data = match coll.find_one(document, None) {
            Ok(d) => match d {
                Some(dd) => dd,
                None => return None
            },
            Err(_) => {
                return None;
            }
        };

        let t: Self::Type = match json::decode(&Bson::Document(data).to_json().to_string()){
            Ok(t) => t,
            Err(_) => return None
        };

        return Some(t);
    }

    fn delete(&self, document: Document) -> bool {
        let coll = DATABASE.collection(self.collection_name());
        match coll.delete_one(document, None) {
            Ok(_) => true,
            Err(_) => false
        }
    }

    fn update(&self) -> bool {
        let coll = DATABASE.collection(self.collection_name());

        let jsn0 = self.to_json();

        let jobj = match jsn0.as_object() {
            Some(j) => j,
            None => return false
        };

        let jsn = jobj;

        let data = {
            let mut doc = Document::new();
            for (key, value) in jsn.clone() {
                doc.insert(key, Bson::from_json(&value));
            }
            doc
        };

        let docu = doc!{"$set" => data};

        match coll.update_one(self.primary(), docu, None) {
            Ok(_) => true,
            Err(_) => false
        }
    }

    fn count(document: Option<Document>) -> i64 {
        let coll = DATABASE.collection(Self::coll_name());
        return match coll.count(document, None) {
            Ok(c) => c,
            Err(_) => 0
        };
    }
}

Here is an example of how it works

use rustc_serialize::json::{ToJson, Json};
use std::collections::BTreeMap;
use bson::Document;

#[derive(RustcDecodable, RustcEncodable, Clone)]
pub struct Test {
    pub id: i64,
    pub name: Option<String>,
}

impl ToJson for Test {
    fn to_json(&self) -> Json {
        let mut test = BTreeMap::new();
        test.insert("id".to_string(), self.id.to_json());
        test.insert("name".to_string(), self.name.to_json());
        Json::Object(test)
    }
}

impl Collections for Test {

    type Type = Test;

    fn collection_name(&self) -> &str {
        "test_col"
    }

    fn coll_name() -> &'static str {
        "test_col"
    }

    fn primary(&self) -> Document {
        let id = self.id.clone();
        doc!{"id" => id}
    }

}

impl Test {

    pub fn new(name: Option<String>) -> Option<Test> {

        let test = Test{
            id: Test::count(None),
            name: name,
        };

        test.insert()
    }

    pub fn from_id(id: i64) -> Option<Test> {
        Test::find_one(Some(doc!{"id" => id}))
    }

    pub fn name(&self) -> Option<String> {
        self.name.clone()
    }

}

While it isnt the very best, it works. I created this small wrapper to make it easy so i am not repeating the same code over.

[lord]

@dariusc93 this is helpful, thanks a bunch! Really appreciate it.

[zonyitoo]

Ah, looks great.

[lord]

Probably not very good code, but this ended up being my serialization/deserialization code with Serde, if anybody else in the future has the same problem I did:

fn decode<T: Deserialize>(doc: &Document) -> Option<T> {
    let doc2 = Bson::Document(doc.clone());
    let mut d = Decoder::new(doc2);
    Deserialize::deserialize(&mut d).ok()
}

fn encode<T: Serialize>(doc: &T) -> Option<Document> {
    let mut e = Encoder::new();
    doc.serialize(&mut e);
    match e.bson() {
        Ok(Bson::Document(d)) => Some(d),
        _ => None,
    }
}

The other things to watch out for is that Serde is going to want to match everything, so you need to make sure to have the Mongo ID structured correctly:

(assuming you're storing a document with one string field 'meow')

#[derive(Clone, Serialize, Deserialize, PartialEq, Debug)]
pub struct MongoID {
    #[serde(rename="$oid")]
    pub oid: String,
}

#[derive(Clone, Serialize, Deserialize, PartialEq, Debug)]
pub struct Test {
    #[serde(rename="_id")]
    pub id: Option<MongoID>,
    pub meow: Option<String>,
}

Again, sorry if off-topic for this issue, but just thought I'd post in case somebody else could benefit from it.

[dariusc93]

When I get time, I may take another look at serde (I do plan on switching in the future, only using rustc_serialize because of the libraries I use support it over serde), but good catch if serde is that strict. Ill give this a try sometime tonight and keep you up to date.

[kyeah]

There's a pair of helper methods that will reduce the boilerplate slightly.

use bson::{from_bson, to_bson};

fn decode<T: Deserialize>(doc: &Document) -> Option<T> {
    from_bson(Bson::Document(doc.clone())).ok()
}

fn encode<T: Serialize>(doc: &T) -> Option<Document> {
    match to_bson(doc) {
        Ok(Bson::Document(d)) => Some(d),
        _ => None,
    }
}

The decode method could possibly be shortened to from_bson(doc).ok() with some changes to the library.

In general Serde is pretty strict, but I've tried to maintain the mapping from structs to Extended JSON as closely as possible when using it. In the meow example, the following will also work, provided you represent the OID as an ObjectId instead of a String:

use bson::oid::ObjectId;

#[derive(Clone, Serialize, Deserialize, PartialEq, Debug)]
pub struct Test {
    #[serde(rename="_id")]
    pub id: Option<ObjectId>,
    pub meow: Option<String>,
}

We should try to get a better README up and host the generated documentation in the near future to make it easier to use the crate and see what's available.

[lord]

Thanks @kyeah, that example was really helpful, couldn't find those helper methods anywhere. README+hosted docs would be fantastic!

[freiguy1]

@kyeah I tried using bson::decoder::from_bson and bson::encoder::to_bson and the compiler yelled at me that they were both private. How is it that I could use those methods like you suggest in your post?

[kyeah]

@freiguy1 small typo -- the encoder and decoder modules are private, but the methods are publically exported in the main library, so bson::from_bson and bson::to_bson should work.

Just fixed this in my previous example. Thanks for pointing it out!

[dariusc93]

@kyeah when using those helpers I normally get "the trait serde::ser::Serialize is not implemented for the type T" Same with Deserialize.

[freiguy1]

I just ran cargo update after not doing so for over a week. Now I get the same errors as @dariusc93. But it looks like it happens in more cases than just using these helpers.

the trait serde::de::Deserialize is not implemented for the type league::db::LeagueDao

but it is:

#[derive(Serialize, Deserialize)]¬
pub struct LeagueDao {¬
    #[serde(rename="_id")]¬
    pub id: u32,¬
    pub name: String,¬
    pub contact_email: String,¬
    pub contact_phone: String,¬
    pub sport: String,¬
    pub teams: Vec<TeamDao>¬
}¬

[dariusc93]

@freiguy1 do you know what version of serde you were on before the update? I normally keep record of the version but since i never previously had it in my project it could be a bug with serde itself. Not just structs but any functions that tries to use those traits seem to have the problem

[dariusc93]

Just noticed serde updated to 0.7 and bson.rs uses 0.6

[freiguy1]

Ahh that might be an issue. @dariusc93: here's my cargo.lock that is successfully building: https://gitlab.com/freiguy1/league-api/blob/master/Cargo.lock

I think the data is in there.

[freiguy1]

@dariusc93 What are you doing to get past this issue? Hardcoding the serde version in your cargo.toml to be something other than the newest?

[dariusc93]

@freiguy1 the issue was on 0.7 so i downgraded to 0.6 but to be safe I did hardcode the versions instead of doing "0.6" to the one in your Cargo.lock. I also noticed you use "*" in your Cargo.toml. I would highly suggest not doing that because you never know what changes to come in the latest that may break your code. As a side note, serde 0.7 is not as strict as 0.6. might be worth taking a look at when things are fixed.

[freiguy1]

Could this get pushed to crates.io?

[zonyitoo]

Will do it after #36 is merged.

[zonyitoo]

Already published. Cheers.

[chaoky]

for future reference,
the crate provides the as_document() method