python - Python：替代 `.replace('null', '""')` 用嵌套 JSON 中的字符串替换 null 和 None 值
问题描述
这是对这个 stackoverflow 问题的跟进
它提供了这个解决方案来替换嵌套 json 中的所有None和值null
r = json.dumps(j).replace('null', '""')
json.loads(r)
虽然这在大多数情况下都有效，但对于某些文件我遇到了这个错误
---------------------------------------------------------------------------
JSONDecodeError Traceback (most recent call last)
<ipython-input-39-9016b2aa3f6e> in <module>()
55 parsed = json.load(f)
56 r = json.dumps(parsed).replace('null', '"NothingExistsHere"' ) #remove all nulls and Nones and replace with empty strings
---> 57 parsed = json.loads(r)
58
59 whichList = testDF.iloc[i]['AbstractHeader']
2 frames
/usr/lib/python3.6/json/__init__.py in loads(s, encoding, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw)
352 parse_int is None and parse_float is None and
353 parse_constant is None and object_pairs_hook is None and not kw):
--> 354 return _default_decoder.decode(s)
355 if cls is None:
356 cls = JSONDecoder
/usr/lib/python3.6/json/decoder.py in decode(self, s, _w)
337
338 """
--> 339 obj, end = self.raw_decode(s, idx=_w(s, 0).end())
340 end = _w(s, end).end()
341 if end != len(s):
/usr/lib/python3.6/json/decoder.py in raw_decode(self, s, idx)
353 """
354 try:
--> 355 obj, end = self.scan_once(s, idx)
356 except StopIteration as err:
357 raise JSONDecodeError("Expecting value", s, err.value) from None
JSONDecodeError: Expecting ',' delimiter: line 1 column 24794 (char 24793)
所以，我想知道是否有替代方法来替换嵌套 JSON 对象中的 null 和 None 。
编辑：
这是一个示例 JSON 对象。我不得不截断它以满足 stackoverflow 的字数限制，因此它可能不包含发生错误的位置。
{
"metadata": {
"abstractText": "Jens Peter Bonde*, Esben Meulengracht Flachs, Susie Rimborg, Clara Copenhagen, DK-2600 Glostrup, Denmark",
"authors": [
"Jens Peter Bonde",
"Esben Meulengracht Flachs",
"Susie Rimborg",
"Clara Helene Glazer",
"Aleksander Giwercman",
"Cecilia H\u00f8st Ramlau-Hansen",
"Karin S\u00f8rig Hougaard",
"Birgit Bjerre H\u00f8yer",
"Katia Keglberg H\u00e6rvig",
"Sesilje Bondo Petersen",
"Lars Rylander",
"Ina Olmer Specht",
"Gunnar Toft",
"Elvira Vaclavik Br\u00e4uner"
],
"creator": "Arbortext Advanced Print Publisher 10.0.1465/W Unicode",
"emails": [
"jens.peter.ellekilde.bonde@regionh.dk",
"sions@oup.com"
],
"referenceMentions": [
{
"context": "hypothesized that a relative excess of estrogen, in particular at the time of testicle differentiation, is a major risk factor for testicular cancer (Henderson et al., 1979).",
"endOffset": 173,
"referenceID": 25,
"startOffset": 149
},
{
"context": "Bootstrapping analyses of the subset of the seven substances with highest breast milk concentrations indicated higher median values in cases but this may be a chance finding since the overall analysis of all compounds was non-significant (Damgaard et al., 2006).",
"endOffset": 261,
"referenceID": 11,
"startOffset": 238
},
{
"context": "The study on brominated flame retardants found increased levels in breast milk samples among cases, but this result was not confirmed in analyses based upon corresponding placenta tissue samples, which a priori were thought to provide the most accurate measure of fetal exposure (Main et al., 2007).",
"endOffset": 298,
"referenceID": 43,
"startOffset": 279
},
{
"context": "In the first study, compounds with short biological half-lives were examined and three out of eight substances were elevated in serum samples compared to controls but these results were not entirely consistent with results based upon urine samples (Choi et al., 2012).",
"endOffset": 267,
"referenceID": 9,
"startOffset": 248
},
{
"context": "The other study examined nine biopersistent pesticides and observed higher concentrations in serum from cases for them all (Shekharyadav et al., 2011).",
"endOffset": 150,
"referenceID": 67,
"startOffset": 123
},
{
"context": "Average sperm counts were almost halved in the exposed group and the effect was most pronounced among men who had been breastfed during childhood (Mocarelli et al., 2011).",
"endOffset": 170,
"referenceID": 52,
"startOffset": 146
},
{
"context": "Most studies measured the contaminants late in pregnancy, at birth or during lactation and not precisely at the time of differentiation of the male gonads, that based upon animal studies in the rat, is assumed to take place in gestational week 8\u201314 (Welsh et al., 2008; Macleod et al., 2010).",
"endOffset": 291,
"referenceID": 88,
"startOffset": 249
},
{
"context": "Most studies measured the contaminants late in pregnancy, at birth or during lactation and not precisely at the time of differentiation of the male gonads, that based upon animal studies in the rat, is assumed to take place in gestational week 8\u201314 (Welsh et al., 2008; Macleod et al., 2010).",
"endOffset": 291,
"referenceID": 41,
"startOffset": 249
},
{
"context": "Considering the long half-lives of the persistent chemicals with little expected fluctuation of tissue levels across a few months, measurements around the time of birth seems a reasonable proxy for exposure levels during early pregnancy although distribution kinetics during pregnancy may be an issue (Verner et al., 2013).",
"endOffset": 322,
"referenceID": 83,
"startOffset": 301
},
{
"context": "Thus, in a study where three blood samples were taken during pregnancy the intra-class coefficient of variation were for most phthalate metabolites between 20% and 40% (Cantonwine et al., 2014).",
"endOffset": 193,
"referenceID": 6,
"startOffset": 168
},
{
"context": "One of the included studies measuring phthalate metabolites in amnion fluid in the second trimester found no association with cryptorchidism and hypospadias even though one DEHP metabolite was related to higher fetal testosterone and lower insulin-like factor III (Jensen et al., 2015).",
"endOffset": 285,
"referenceID": 28,
"startOffset": 264
},
{
"context": "It is therefore noteworthy that knowledge regarding the transport of persistent xenobiotics over the placenta to the fetus has been known for decades (Needham et al., 1999; Mazdai et al., 2003).",
"endOffset": 193,
"referenceID": 54,
"startOffset": 150
},
{
"context": "It is therefore noteworthy that knowledge regarding the transport of persistent xenobiotics over the placenta to the fetus has been known for decades (Needham et al., 1999; Mazdai et al., 2003).",
"endOffset": 193,
"referenceID": 45,
"startOffset": 150
},
{
"context": "More recent findings indicate that rapidly metabolized compounds also pass through the placenta and these have been detected in cord blood and amnion fluid (Jensen et al., 2015).",
"endOffset": 177,
"referenceID": 28,
"startOffset": 156
},
{
"context": "However, the studies included in this review encompass studies based upon samples biobanked >50 years ago such as the Collaborative Perinatal Study with relatively high serum concentrations (Longnecker et al., 2002; Pierik et al., 2007; McGlynn et al., 2008, 2009b; Trabert et al., 2012) as well as studies based upon sampling much later (Damgaard et al.",
"endOffset": 287,
"referenceID": 39,
"startOffset": 190
},
{
"context": "However, the studies included in this review encompass studies based upon samples biobanked >50 years ago such as the Collaborative Perinatal Study with relatively high serum concentrations (Longnecker et al., 2002; Pierik et al., 2007; McGlynn et al., 2008, 2009b; Trabert et al., 2012) as well as studies based upon sampling much later (Damgaard et al.",
"endOffset": 287,
"referenceID": 57,
"startOffset": 190
},
{
"context": "However, the studies included in this review encompass studies based upon samples biobanked >50 years ago such as the Collaborative Perinatal Study with relatively high serum concentrations (Longnecker et al., 2002; Pierik et al., 2007; McGlynn et al., 2008, 2009b; Trabert et al., 2012) as well as studies based upon sampling much later (Damgaard et al.",
"endOffset": 287,
"referenceID": 80,
"startOffset": 190
},
{
"context": ", 2012) as well as studies based upon sampling much later (Damgaard et al., 2006; Main et al., 2007; Virtanen et al., 2012).",
"endOffset": 123,
"referenceID": 11,
"startOffset": 58
},
{
"context": ", 2012) as well as studies based upon sampling much later (Damgaard et al., 2006; Main et al., 2007; Virtanen et al., 2012).",
"endOffset": 123,
"referenceID": 43,
"startOffset": 58
},
{
"context": ", 2012) as well as studies based upon sampling much later (Damgaard et al., 2006; Main et al., 2007; Virtanen et al., 2012).",
"endOffset": 123,
"referenceID": 87,
"startOffset": 58
},
{
"context": "We analyzed the male reproductive disorders, with the exception of low sperm count where no risk estimates were available, as one entity with reference to the testicular dysgenesis syndrome hypothesis on shared etiology of the four male reproductive disorders (Skakkebaek, 2002) and with reference to experimental evidence suggesting multiple mechanistic pathways of endocrine disruption (Toppari, 2002).",
"endOffset": 403,
"referenceID": 78,
"startOffset": 388
},
{
"context": "Although focus was on developmental disorders induced by chemical exposure during critical time windows in early life (Lee and Jacobs, 2015), there only exists one small study using prenatal or postnatal bio-specimens to assess testicular cancer (Cohn et al., 2010).",
"endOffset": 265,
"referenceID": 10,
"startOffset": 246
},
{
"context": "First of all, the biological half-lives of many of these compounds are counted in decades (Hagmar et al., 2006).",
"endOffset": 111,
"referenceID": 21,
"startOffset": 90
},
{
"context": "Another study found high correlations of persistent chemical levels in serum samples in adult men obtained up to 10 years apart (Hagmar et al., 2006).",
"endOffset": 149,
"referenceID": 21,
"startOffset": 128
},
{
"context": ", 2011), only three human studies have to date linked prenatal and postnatal exposure to xenobiotics with sperm count in adults (Mocarelli et al., 2011; Axelsson et al., 2015; Vested et al., 2013).",
"endOffset": 196,
"referenceID": 52,
"startOffset": 128
},
{
"context": "Some in vitro and in vivo animal studies have demonstrated combined effects of endocrine disrupting chemicals at levels at which the individual chemicals do not induce observable effects (Kortenkamp, 2007; Kortenkamp, 2014).",
"endOffset": 223,
"referenceID": 31,
"startOffset": 187
},
{
"context": "Some in vitro and in vivo animal studies have demonstrated combined effects of endocrine disrupting chemicals at levels at which the individual chemicals do not induce observable effects (Kortenkamp, 2007; Kortenkamp, 2014).",
"endOffset": 223,
"referenceID": 32,
"startOffset": 187
},
{
"context": "Another concern is the possibility of non-monotonic dose\u2013response relationships which may explain apparently inconsistent findings across studies because of different levels and ranges of exposures (Vandenberg et al., 2012).",
"endOffset": 223,
"referenceID": 82,
"startOffset": 198
},
{
"context": "In this light, estimates of the burden of disease and costs of exposure to endocrine chemicals (Trasande et al., 2016) seem highly speculative, at least with respect to male reproductive disorders.",
"endOffset": 118,
"referenceID": 81,
"startOffset": 95
}
],
"references": [
{
"author": [
"TE Arbuckle",
"M Fisher",
"S Macpherson",
"C Lang",
"G Provencher",
"A LeBlanc",
"Hauser"
],
"citeRegEx": "Arbuckle et al\\.,? \\Q2020\\E",
"shortCiteRegEx": "Arbuckle et al\\.",
"title": "The review was funded by InterReq ReproUnion (NYPS 20200407)",
"venue": "Conflict of interest None declared",
"year": 2020
},
{
"author": [
"Reprod",
"21:657\u2013665. Barr DB",
"Weihe P",
"Davis MD",
"Needham LL",
"Grandjean P"
],
"citeRegEx": "2006 and P.,? \\Q2006\\E",
"shortCiteRegEx": "2006 and P.",
"title": "Serum polychlorinated",
"venue": null,
"year": 2006
},
{
"author": [
"Skakkebaek NE"
],
"citeRegEx": "NE.,? \\Q2003\\E",
"shortCiteRegEx": "NE.",
"title": "Testicular dysgenesis syndrome",
"venue": "Horm Res",
"year": 2003
},
{
"author": [
"HE Virtanen",
"JJ Koskenniemi",
"E Sundqvist",
"KM Main",
"H Kiviranta",
"JT Tuomisto",
"J Tuomisto",
"M Viluksela",
"T Vartiainen",
"NE Skakkebaek"
],
"citeRegEx": "Virtanen et al\\.,? \\Q2012\\E",
"shortCiteRegEx": "Virtanen et al\\.",
"title": "Associations between congenital cryptorchidism in newborn boys and levels of dioxins and PCBs in placenta",
"venue": "Int J Androl 2012;35:283\u2013293",
"year": 2012
},
{
"author": [
"M Welsh",
"PT Saunders",
"M Fisken",
"HM Scott",
"GR Hutchison",
"LB Smith",
"RM. Sharpe"
],
"citeRegEx": "Welsh et al\\.,? \\Q2008\\E",
"shortCiteRegEx": "Welsh et al\\.",
"title": "Identification in rats of a programming window for reproductive tract masculinization, disruption of which leads to hypospadias and cryptorchidism",
"venue": "J Clin Invest 2008;118:1479\u20131490",
"year": 2008
},
{
"author": [
"SS White",
"SE Fenton",
"EP. Hines"
],
"citeRegEx": "White et al\\.,? \\Q2011\\E",
"shortCiteRegEx": "White et al\\.",
"title": "Endocrine disrupting properties of perfluorooctanoic acid",
"venue": "J Steroid Biochem Mol Biol",
"year": 2011
},
{
"author": [
"J Zhang",
"L Liu",
"X Wang",
"Q Huang",
"M Tian",
"H. Shen"
],
"citeRegEx": "Zhang et al\\.,? \\Q2016\\E",
"shortCiteRegEx": "Zhang et al\\.",
"title": "Low-level environmental phthalate exposure associates with urine metabolome alteration in a Chinese Male Cohort",
"venue": null,
"year": 2016
}
],
"sections": [
{
"heading": null,
"text": "..........................................................................................................................."
},
{
"heading": "Methods",
"text": "The review was conducted and reported in accordance with the MOOSE guidelines for Meta-analyses and Systematic reviews of Observational Studies (Stroup et al., 2000)."
},
{
"heading": "Protocol and registration",
"text": "A review protocol was registered at PROSPERO.org with registration number CRD4201603742 prior to initiation of the review process, on 12 April 2016 with amendments on 12 May 2016 (CRD420160374X). The amendments specified the four main hypotheses, provided details on sensitivity analyses and discarded the initial idea to analyze data according to estrogenic or anti-androgenic activity of measured compounds which proved non-feasible. The protocol was updated before the review process and data analysis were initiated."
},
{
"heading": "Information sources",
"text": "The databases PubMed and Embase were used as they cover the vast majority of relevant journals for the subject."
},
{
"heading": "Eligibility criteria",
"text": "We conducted a systematic search of original peer-reviewed original papers in English published between 1966 and 12 April 2016 to identify\n106 Bonde et al.\njournal articles providing quantitative data on the association between xenobiotic endocrine disrupting chemicals and male reproductive disorders in humans. The complete l n quality other than sperm concentration. (3) Studies based upon chemical analysis of exposure after puberty except studies of testicular cancer. (4) Studies repeating risk estimates reported in previous publications for example, studies addressing risk according to gene polymorphisms for substances, where risk estimates for the entire population were provided earlier. (5) Ecological studies with exposure information at the population level rather than at the individual level."
},
{
"heading": "Search and study selection",
"text": "We combined medical subject headings and generic terms for the made to retrieve papers from the unpublished literature. The steps in the literature search are displayed in Fig. 1."
},
{
"heading": "Data extraction",
"text": "Descriptive information (Tables II\u2013V) was recorded from each publication using a standardized form constructed prior to the collection theoretical inverse relationships. If the relevant relative risks were not reported but data were available, the authors of this paper computed risk estimates and CIs."
},
{
"heading": "Quality assessment",
"text": ""
},
{
"heading": "Reporting",
"text": "Each publication was evaluated for completeness of reporting of the following 11 study characteristics modified after Bonzini et al., (2007): (i) study design, (ii) scores for each paper was \u22658 (Bonzini et al., 2007). Completeness of reporting is not a direct measure of quality but high completeness is needed for adequate evaluation of bias and confounding."
},
{
"heading": "Bias and confounding",
"text": "The decision to limit the review to observational epidemiological studies applying biological monitoring for exposure assessment is considered of major importance to counteract biased findings because of differential recall of exposure often introduced higher risk of bias if two or more of these sources of bias were present.\nRisk of bias across studies was evaluated by means of funnel plots to assess risk of publication bias."
},
{
"heading": "Meta-analysis",
"text": "We applied a stepwise approach addressing four different scenarios for the association between chemical exposure and included outcomes with increasing level of specificity. Only cryptorchidism, hypospadias and testicular cancer were included in Steps testicular dysgenesis hypothesis stating that all male reproductive outcomes at least partly are manifestations of the same underlying causal agents operating in early life (Skakkebaek et al., 2001; .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. T ab le V I S tu di es re po rt in g as so ci at io ns be tw ee n pr en at al ex po su re an d sp er m co un t w it ho ut ri sk es ti m at es (n ot el ig ib le fo r m et aan al ys is ). R ef er en ce L oc at io n S tu dy po pu la ti on S tu dy de si gn N po pu la ti on B io sp ec im en M ea su re d xe no bi ot ic s R es ul ts C R a B ia sb M oc ar el li et al .( 20 11 ) Ita ly , Se ve so Ex po se d m en ar e so ns of w om en liv in g ne ar th e Se ve so ch em ic al pl an t at th e tim e of th e ac ci de nt in 19 76 .C on tr ol s w er e bl oo d do no rs w ho se m ot he rs w er e no t ex po se d M ix ed de si gn 39 ex po se d /5 8 re fe re nt s M at er na ls er um in ex po se d m en .R ef er en ts ha d an as su m ed ex po su re of 10 pp t Se ru m sa m pl es T C D D A ve ra ge sp er m co un ts w er e al m os t ha lv ed in th e ex po se d gr ou p an d th e ef fe ct w as m os t pr on ou nc ed am on g br ea st fe d m en 8 1 V es te d et al . (2 01 3) D en m ar k Pr eg na nc y co ho rt ,m al e of fs pr in g Fo llo w -u p 16 9 M at er na ls er um PF O S PF O A bu t no t PF O S w as as so ci at ed w ith lo w er ad ju st ed sp er m co nc en tr at io n (p -t re nd = 0. 01 ) 11 0 PF O A V es te d et al . (2 01 4) D en m ar k Pr eg na nc y co ho rt ,m al e of fs pr in g Fo llo w -u p 17 3 M at er na ls er um PC B (s ix co ng en er s) Su m PC B an d p, p\u2032 -D D E w as no t as so ci at ed w ith sp er m co un t 11 0 D D E A xe ls so n et al .( 20 15 ) Sw ed en M ai nl y yo un g m en pr es en tin g fo r m ili ta ry he al th bo ar d bu t al so a fe w fr om th e ge ne ra lp op ul at io n (t hr ou gh an no un ce m en ts in sc ho ol s) Fo llo w -u p 11 2 M at er na ls er um T hr ee D EH P m et ab ol ite s T hr ee D iN P m et ab ol ite s N on e of th e si x ph th al at e m et ab ol ite s w er e as so ci at ed w ith Sp er m co un t 7 1 a C R :c om pl et en es s of re po rt in g on a sc al e fr om 0 (lo w co m pl et en es s) to 11 (h ig h co m pl et en es s) . b Bi as an d co nf ou nd in g: 1 = hi gh er ris k of bi as ,0 = lo w er ris k of bi as .\n113Review on EDC and male reproduction\nSkakkebaek, 2003, 2004). One study reported multiple PCB and PBDE congener specific risk estimates but no summary estimate (Carmichael et al., 2010). For this study, we selected the PCB-153 and PBDE-100 as analyses were performed using R and a significance level of 0.05. We also used this software to create funnel plots of the standard error by the logarithm of the RR separately for each of the exposures and for the higher quality studies, and inspected these plots for evidence of publication bias.\n114 Bonde et al."
},
{
"heading": "Results",
"text": ""
},
{
"heading": "The study base",
"text": "We identified 33 papers that were based upon 28 independent studies reported 89 associations between measured levels of endocrine disrupting chemicals and 2015).\n115Review on EDC and male reproduction"
},
{
"heading": "Authors\u2019 roles",
"text": "Jens Peter Bonde and Elvira Bra\u00fcner conceived and organized the study, sifted titles and abstracts and independently included and excluded papers. Esben the manuscript to which all authors contributed and approved the final version format."
},
{
"heading": "Funding",
"text": "The review was funded by InterReq ReproUnion (NYPS 20200407)."
},
{
"heading": "Conflict of interest",
"text": "None declared."
}
],
"source": "CRF",
"title": "The epidemiologic evidence linking prenatal and postnatal exposure to endocrine disrupting chemicals with male reproductive disorders: a systematic review and meta-analysis",
"year": 2016
},
"name": "http.__pdfs.semanticscholar.org_9103_df68cb38538756b1faf78a2d991b3f6f11b3.pdf"
}
解决方案
对分层数据进行结构不敏感的字符串操作是一个坏主意- 您应该从核心解决问题，在您的情况下是 JSON 编码。遗憾的是，内置json模块不允许轻松覆盖已知结构的默认行为，None它就是其中之一。除非您想深入并覆盖json.JSONEncoder.iterencode()（及其内部_iterencode_*方法）以迫使它屈服于您的意愿，或者使用替代/较慢的 JSON 编码器，否则最好的选择是预处理您的数据。
这比你想象的要简单得多：
import json
from collections.abc import Mapping, Iterable
def replace_none(o, repl=''):
if o is None:
return repl
if isinstance(o, Mapping):
return {k: replace_none(v, repl) for k, v in o.items()}
elif not isinstance(o, str) and isinstance(o, Iterable):
return [replace_none(v, repl) for v in o]
return o
# and to test it:
data = {'this': {'that': 'and', 'the': ['other thing', None]}}
r = json.dumps(replace_none(data))
# {"this": {"that": "and", "the": ["other thing", ""]}}
推荐阅读
- javascript - Express 中间件混淆
- react-native - 为什么 react-native 需要很长时间才能启动？
- spring-boot - 单元测试 Camel/RabbitMQ 路由问题
- arrays - 使用 Coldfusion 从多个结构和数组中获取值
- postgresql - postgres窗口函数三倍查询时间
- c++ - 使用 SDL2 时，'' 的左侧必须有类/结构/联合
- python - python中的Tkinter - 未找到
- python - 如何使 dockerized Flask 应用程序将输出写入文件？
- javascript - 如何从 NodeJS 连接到 PostgreSQL
- laravel - Laravel - 从控制器中的多个字段计算最小值