Posts

Showing posts with the label Mongodb

Bulk Upsert In MongoDB Using Mongoose

Answer : Not in "mongoose" specifically, or at least not yet as of writing. The MongoDB shell as of the 2.6 release actually uses the "Bulk operations API" "under the hood" as it were for all of the general helper methods. In it's implementation, it tries to do this first, and if an older version server is detected then there is a "fallback" to the legacy implementation. All of the mongoose methods "currently" use the "legacy" implementation or the write concern response and the basic legacy methods. But there is a .collection accessor from any given mongoose model that essentially accesses the "collection object" from the underlying "node native driver" on which mongoose is implemented itself: var mongoose = require('mongoose'), Schema = mongoose.Schema; mongoose.connect('mongodb://localhost/test'); var sampleSchema = new Schema({},{ "strict": false }); va...

Bulk Update In Pymongo Using Multiple ObjectId

Answer : Iterate through the id list using a for loop and send the bulk updates in batches of 500: bulk = db.testdata.initialize_unordered_bulk_op() counter = 0 for id in ids: # process in bulk bulk.find({ '_id': id }).update({ '$set': { 'isBad': 'N' } }) counter += 1 if (counter % 500 == 0): bulk.execute() bulk = db.testdata.initialize_ordered_bulk_op() if (counter % 500 != 0): bulk.execute() Because write commands can accept no more than 1000 operations (from the docs ), you will have to split bulk operations into multiple batches, in this case you can choose an arbitrary batch size of up to 1000. The reason for choosing 500 is to ensure that the sum of the associated document from the Bulk.find() and the update document is less than or equal to the maximum BSON document size even though there is no there is no guarantee using the default 1000 operations requests will fit under the 16MB BSON limit. Th...

Convert Milliseconds To Date In Mongodb Aggregation Pipeline For Group By?

Answer : I'm trying to get the logic behind converting the txnTime field to a date object because grouping by either a date field or a timestamp in milliseconds (like what you are presently doing) will yield the same result as they both are unique in their respective formats! To change the txnTime field to a date object you should then include a $project pipeline before the $group pipeline stage with this expression "txnTime": { "$add": [ new Date(0), "$txnTime" ] } so that you can do your $group operation on the converted/projected txnTime field: var convertedTxnTime = { "$add": [new Date(0), "$txnTime"] }; /* If using MongoDB 4.0 and newer, use $toDate var convertedTxnTime = { "$toDate": "$txnTime" }; or $convert var convertedTxnTime = { "$convert": { "input": "$txnTime", "to": "date" } }; */ db.campaign_wallet.aggregate([ { "$match...

Aggregate/project Sub-document As Top-level Document In Mongo

Answer : When you have many, many fields in the sub-document and occasionally it is updated with new fields, then projection is not a viable option. Fortunately, since 3.4, MongoDB has a new operator called $replaceRoot . All you have to do is add a new stage at the end of your pipeline. db.getCollection('sample').aggregate([ { $replaceRoot: {newRoot: "$command"} }, { $project: {score: 0 } //exclude score field } ]) This would give you the desired output. Note that in case of aggregation (especially after a $group stage) the 'command' document could be an array and could contain multiple documents. In this case you need to $unwind the array first to be able to use $replaceRoot . As you have guessed, $project allows you to do that: db.col.aggregate([ { $project : { _id: "$command._id", name: "$command.name", strike: "$command.strike", duratio...

Can't Use Mongo Command, Shows Command Not Found On Mac

Answer : You need to add the path to "mongo" to your terminal shell. export PATH=$PATH:/usr/local/mongodb/bin Did you do the last step with paths.d? If so, try restarting your terminals. Do you have a good reason for using 1.8.5? The current stable is 2.0.4, and it has many useful upgrades from 1.8.x You'll have to add the location of the Mongo binary to PATH. Follow the steps below in order to make the PATH variable permanent: Open Terminal and navigate to your user directory. Run touch ~/.bash_profile and then open ~/.bash_profile . In TextEdit, add export PATH="<mongo-directory>/bin:$PATH" (Keep the quote marks - related to white spaces). Save the .bash_profile file and Quit (Command + Q) Text Edit. Run source ~/.bash_profile . Run echo $PATH and check if the you see that the Mongo binary was added. (*) Notice that the PATH variable is now available only for the current terminal and not to processes that were already started...

Converting A Mongo Stored Date Back Into Milliseconds Since Unix Epoch When Loaded?

Answer : You can add the numerical milliseconds version of timestamp as a virtual attribute on the schema: schema.virtual('timestamp_ms').get(function() { return this.timestamp.getTime(); }); Then you can enable the virtual field's inclusion in toObject calls on model instances via an option on your schema: var schema = new Schema({ timestamp: Date }, { toObject: { getters: true } }); var schema = new Schema({ timestamp: {type:Number, default: new Date().getTime()} }); Hope this will solve your issue. As a best practice, I would say: keep your data the type it deserves . Anyway, if your client needs to treat with numbers, you can simply pass the date as milliseconds to the client, and still work with Date objects in Node. Just call timestamp.getTime() and ta-da, you have your unix timestamp ready for the client.

Creating BSON Object From JSON String

Answer : ... And, since 3.0.0, you can: import org.bson.Document; final Document doc = new Document("myKey", "myValue"); final String jsonString = doc.toJson(); final Document doc = Document.parse(jsonString); Official docs: Document.parse(String) Document.toJson() Official MongoDB Java Driver comes with utility methods for parsing JSON to BSON and serializing BSON to JSON. import com.mongodb.DBObject; import com.mongodb.util.JSON; DBObject dbObj = ... ; String json = JSON.serialize( dbObj ); DBObject bson = ( DBObject ) JSON.parse( json ); The driver can be found here: https://mongodb.github.io/mongo-java-driver/ The easiest way seems to be to use a JSON library to parse the JSON strings into a Map and then use the putAll method to put those values into a BSONObject . This answer shows how to use Jackson to parse a JSON string into a Map .

Connections % Of Configured Limit Has Gone Above 80 From Mongo Atlas

Answer : removing all IP address and waiting 5 minutes works also for me . seems like it kills all opened connections. Don't forget to allow your ip after that see my cluster connections there is an opened issue with mongoose. it might be the root cause https://github.com/Automattic/mongoose/issues/8059 I resolved it by deleting all IP whitelist and wait for 5 minutes. We can monitor that, the connections are decreasing in mongo atlas cluster. At last, it became Zero. Then added IP whitelist from anywhere to access(Not secure. Just to work. Or whitelist current IP and server Ip). Its work fine.

Can Mongo Upsert Array Data?

Answer : I'm not aware of an option that would upsert into an embedded array as at MongoDB 2.2, so you will likely have to handle this in your application code. Given that you want to treat the embedded array as sort of a virtual collection, you may want to consider modelling the array as a separate collection instead. You can't do an upsert based on a field value within an embedded array, but you could use $addToSet to insert an embedded document if it doesn't exist already: db.soup.update({ "tester":"tom" }, { $addToSet: { 'array': { "id": "3", "letter": "d" } } }) That doesn't fit your exact use case of matching by id of the array element, but may be useful if you know the expected current value. I just ran into this problem myself. I wasn't able to find a one-call solution, but I found a two-call solution that works when you have ...

Connect Robomongo To MongoDB Docker Container

Answer : There is another way. You can SSH with Robomongo into your actual virtual server that hosts your docker applications (SSH tab, check "Use SSH tunnel" and complete the other fields accordingly) Now ssh into the same machine in your terminal. docker ps should show you your MongoDB container. docker inspect <mongo container id> will print out complete information about that container. Look for IPAddress in the end, that will give you the local IP of the container. In the "Connection" tab in Robomongo use that container IP to connect. Another sidenote: Make sure that you don't expose your mongodb service ports in any way (neither Dockerfile nor docker-compose.yml), cause that will make your database openly accessible from everywhere. Assuming that you don't have set up a username / password for that service you will be scanned and hacked soon. The easiest way is to enable forwarding the Mongo Container itself, here's how my docker-compo...

C# MongoDB Distinct Query Syntax

Answer : You could try the following approach: var filter = new BsonDocument(); var categoriesList = await blogContext.Articles.DistinctAsync<string>("categories", filter);

Conditional $sum In MongoDB

Answer : As Sammaye suggested, you need to use the $cond aggregation projection operator to do this: db.Sentiments.aggregate( { $project: { _id: 0, Company: 1, PosSentiment: {$cond: [{$gt: ['$Sentiment', 0]}, '$Sentiment', 0]}, NegSentiment: {$cond: [{$lt: ['$Sentiment', 0]}, '$Sentiment', 0]} }}, { $group: { _id: "$Company", SumPosSentiment: {$sum: '$PosSentiment'}, SumNegSentiment: {$sum: '$NegSentiment'} }}); Starting from version 3.4, we can use the $switch operator which allows logical condition processing in the $group stage. Of course we still need to use the $sum accumulator to return the sum. db.Sentiments.aggregate( [ { "$group": { "_id": "$Company", "SumPosSenti": { "$sum": { "$switch": { ...

Can A $text Search Perform A Partial Match

Answer : MongoDB $text searches do not support partial matching. MongoDB allows text search queries on string content with support for case insensitivity, delimiters, stop words and stemming. And the terms in your search string are, by default, OR'ed. Taking your (very useful :) examples one by one: SINGLE TERM, PARTIAL // returns nothing because there is no world word with the value `Crai` in your // text index and there is no whole word for which `Crai` is a recognised stem db.submissions.find({"$text":{"$search":"\"Crai\""}}) MULTIPLE TERMS, COMPLETE // returns the document because it contains all of these words // note in the text index Dr. Bob is not a single entry since "." is a delimiter db.submissions.find({"$text":{"$search":"\"Craig\" \"Dr. Bob\""}}) MULTIPLE TERMS, ONE PARTIAL // returns the document because it contains the whole word "Craig" an...

Aggregation With Update In MongoDB

Answer : After a lot of trouble, experimenting mongo shell I've finally got a solution to my question. Psudocode: # To get the list of customer whose score is greater than 2000 cust_to_clear=db.col.aggregate( {$match:{$or:[{status:'A'},{status:'B'}]}}, {$group:{_id:'$cust_id',total:{$sum:'$score'}}}, {$match:{total:{$gt:500}}}) # To loop through the result fetched from above code and update the clear cust_to_clear.result.forEach ( function(x) { db.col.update({cust_id:x._id},{$set:{clear:'Yes'}},{multi:true}); } ) Please comment, if you have any different solution for the same question. With Mongo 4.2 it is now possible to do this using update with aggregation pipeline. The example 2 has example how you do conditional updates: db.runCommand( { update: "students", updates: [ { q: { }, u: [ { $set: { average : { $avg: "$tests...