7.8 数据处理
最后更新于：2022-04-01 23:34:50
# 数据处理
## csv处理
* i-csv
* import-csv
* export-csv
* json2csv
* fastcsv
### json2csv
json to csv converter
[https://github.com/i5ting/json2csv](https://github.com/i5ting/json2csv)
## import-csv
[https://github.com/i5ting/import-csv](https://github.com/i5ting/import-csv)
默认使用gbk编码
~~~
var import_csv = require('import-csv')
import_csv('test.csv',function(err, data){
console.log(data);
})
~~~
指定字符编码机
~~~
var import_csv = require('import-csv')
import_csv('test.csv',function(err, data){
console.log(data);
}, 'gbk')
~~~
## export-csv
[https://github.com/i5ting/export-csv](https://github.com/i5ting/export-csv)
默认使用gbk编码
~~~
var export_csv = require('.')
var data = [
{a:1,b:2},
{a:2,b:2},
{a:3,b:2}
]
export_csv(data, 'test1.csv')
~~~
过滤item，重写数据
~~~
var export_csv = require('.')
var data = [
{ a: 1, b: 2 },
{ a: 2, b: 2 },
{ a: 3, b: 2 }
]
export_csv(data, 'test3.csv', function (item) {
for (var key in item) {
item[key] = 'yy +' + item[key];
}
return item;
})
~~~
指定回调函数
~~~
var export_csv = require('.')
var data = [
{a:1,b:2},
{a:2,b:2},
{a:3,b:2}
]
export_csv(data, 'test2.csv', function (item) {
return item;
}, function () {
console.log('end...');
})
~~~
指定回调函数，并生成header
~~~
var export_csv = require('.')
var data = [
{a:1,b:2},
{a:2,b:2},
{a:3,b:2}
]
export_csv(data, 'test2.csv', function (item) {
return item;
}, function () {
console.log('end...');
}, true)
~~~
## mongodb如何统计
~~~
var json2csv = require('json2csv');
require('../db.js');
var fs = require('fs');
var Order = require('../app/models/order');
var Activity = require('../app/models/activity');
var Wechat = require('../app/models/wechat');
var Contact = require('../app/models/contact');
function main(){
Order.model.aggregate([
{$match : {status2 : {$exists : true} }},
{$group : {_id : "$activity" , count : {$sum : "$product_count"}}}
]).exec(function (err, data) {
var obj = [];
for (var i = data.length - 1; i >= 0; i--) {
var json = {
"url" : "http://shop.mengxiaoban.cn/iscroll.html?id=" + String(data[i]._id),
"count" : String(data[i].count)
}
obj.push(json);
console.log(json.url);
}
var opts = {
data: obj,
fields: ['url','count'],
fieldNames: ['url','count'],
quotes: ''
};
json2csv(opts, function(err, csv) {
if (err) console.log(err);
fs.writeFile('./1119todayurl.csv', csv, function(err) {
if (err) throw err;
console.log('file saved');
});
});
})
}
}
main();
~~~
AGGREGATION 关联
《SQL to Aggregation Mapping Chart》
[http://docs.mongodb.org/manual/reference/sql-aggregation-comparison/](http://docs.mongodb.org/manual/reference/sql-aggregation-comparison/)
## 优化查询速度
了解索引优化
~~~
ContactSchema = new Schema({
...
owner: {
type: Schema.ObjectId,
required: true,
index: true
}
});
~~~
也可以这样的
~~~
ContactSchema.ensureIndexes(owner);
~~~
## 了解explain
~~~
db.usermodels.find({
'_id' :{
"$gt" :ObjectId("55940ae59c39572851075bfd")
}
}).explain()
~~~
关注点
* stage：查询策略
* nReturned：返回的文档行数
* needTime：耗时（毫秒）
* indexBounds：所用的索引
[http://docs.mongodb.org/v3.0/reference/explain-results/](http://docs.mongodb.org/v3.0/reference/explain-results/)
## 了解profile
profile级别有三种：
* 0：不开启
* 1：记录慢命令，默认为大于100ms
* 2：记录所有命令
* 3、查询profiling记录 开启
~~~
db.setProfilingLevel(2, 20)
~~~
默认记录在system.profile中
~~~
db['system.profile'].find()
~~~
## MONGODB的部署
* replset
* shard
我写的《 mongodb运维之副本集实践》
[https://cnodejs.org/topic/5590adbbebf9c92d17e734de](https://cnodejs.org/topic/5590adbbebf9c92d17e734de)
';