|
|
|
|
Genome Res. 15:848-855, 2005 ©2005 by Cold Spring Harbor Laboratory Press; ISSN 1088-9051/05 $5.00 Methods Discovering functional transcription-factor combinations in the human cell cycleDepartment of Genetics, Harvard Medical School, Boston, Massachusetts 02115, USA
With the completion of full genome sequences and advancement in high-throughput technologies, in silico methods have been successfully used to integrate diverse data sources toward unraveling the combinatorial nature of transcriptional regulation. So far, almost all of these studies are restricted to lower eukaryotes such as budding yeast. We describe here a computational search for functional transcription-factor (TF) combinations using phylogenetically conserved sequences and microarray-based expression data. Taking into account both orientational and positional constraints, we investigated the overrepresentation of binding sites in the vicinity of one another and whether these combinations result in more coherent expression profiles. Without any prior biological knowledge, the search led to the discovery of several experimentally established TF associations, as well as some novel ones. In particular, we identified a regulatory module controlling cell cycle-dependent transcription of G2-M genes and expanded its functional generality. We also detected many homotypic combinations, supporting the importance of binding-site density in transcriptional regulation of higher eukaryotes.
Cis-regulation of gene expression by the binding of transcription factors (TFs) is a critical component of cellular physiology. In eukaryotes, a battery of TFs often work together in a combinatorial fashion to enable cells to respond to a wide spectrum of environmental and developmental signals. Integration of genome sequences and/or ChIP-chip data with gene-expression data has facilitated in silico discovery of how the combinatorics and positioning of TF-binding sites underlie gene activation in a variety of cellular processes for relatively simple organisms such as Saccharomyces cerevisiae and Caenorhabditis elegans (Bussemaker et al. 2001
As the functional interactions between TFs often require them to be in physical proximity, their binding sites are likely to be overrepresented in the vicinity of each other. Exploiting such property, we devised a two-step strategy (Fig. 1) to reveal known or novel transcription factors that work in concert. Starting from a TF-binding site of interest, our algorithm first discovers significantly enriched neighboring motif(s) using human-mouse conserved sequences, and then examines the functional significance of their physical proximity through the assessment of similarity in expression profiles. Applying this methodology to human promoter sequences and a cell cycle expression data set, we found a number of motif pairs that not only preferentially co-occur nearby, but also appear to act together in determining gene expression pattern, including a G2-M regulatory module. In addition to heterotypic interactions, we observed a homotypic distribution of transcription-factor binding sites, as many of them are specifically enriched around themselves.
Extraction of human promoter sequence and phylogenetic footprinting We had previously mapped UniGene clusters onto the human genome as well as generated a "mousenized" version of the genome (http://club.med.harvard.edu/hummus/hummus.html). To build a promoter sequences set, we extracted the sequence 1 kb upstream of 11,436 curated RefSeq mRNAs as putative promoter regions. While some regulatory elements can act over very large distances, up to several kilobases from transcriptional start sites (TSS), we focused on sequences in the relative proximity of TSS, as they are most likely to contain regulatory information for evolutionarily conserved biological processes such as the cell cycle.
Accuracy of identifying binding sites by weight matrix
Significantly enriched neighbor motifs To search for enriched neighbor motifs, we extracted both human and corresponding mouse sequences in the vicinity (e.g., 50 and 100 bp) of the conserved anchor motif sites. As the functional interactions between TFs often impose orientational constraints, upstream and downstream sequences were grouped separately. A blind and systematic search was then conducted for shared sequence features with the program AlignACE (Roth et al. 1998
We selected the most statistically significant neighbor motifs using a measure called neighbor specificity (NS) score. It quantifies how specific a neighbor motif targets the neighboring region of the anchor motif, given its rate of occurrence in all promoters. To reduce the bias in assessing the degree of specificity, the calculation was performed based on statistics over the entire 1-kb human sequences, including conserved as well as nonconserved regions. We corrected for multiple testing by calculating NS score cutoffs corresponding to an FDR (Storey and Tibshirani 2003
Homotypic distribution of TF-binding sites
Heterotypic neighbor motifs It is conceivable that some neighbor motifs may happen to be enriched in the vicinity of anchor motifs simply because they both prefer the same location relative to TSS, but have nothing to do with each other. To filter out such potential positionally biased scenarios, for each statistically overrepresented hetero-neighbor motif (i.e., anchor neighbor), we randomly selected the same number of promoters as those with its parent anchor motif and extracted segments of same window size from the same distance upstream of TSS as those containing anchor motif, followed by identical motif search procedures. The random sampling process was repeated 100 times, and we rejected the neighbor motif if it was "rediscovered" by any of these runs. After applying such a positional bias filter, we ended up with 636 (852) significant hetero-neighbor motifs from a 50-bp (100-bp) window, 40 (37) of which could be mapped to known TRANSFAC matrices (CompareACE > 0.85).
Functional anchor-neighbor motif combinations Given the specific enrichment of neighbor motifs, we next asked whether some of them may functionally interact with their corresponding anchors by analyzing a human cell cycle expression data set (Whitfield et al. 2002
Our approach identified several experimentally established associations between TFs. For instance, the cooperation between E2F and NF-Y, two main regulators of cell cycle, has been well documented (van Ginkel et al. 1997
A module controlling transcription of G2-M genes
Most of the genes containing the NF-Y-CDE-CHR combination are indeed cell cycle periodic and peak in G2-M phases. We identified 20 genes with the module in their 1-kb promoter sequences, 17 of which are included in the cell cycle expression data set. Based on their microarray data set, Whitfield et al. (2002
Combinations enriched in other expression clusters We also looked for TF modules that may regulate genes of other phases of the cell cycle. E2F has a well-established role in controlling G1/S transition. We found two E2F combinations with GC-rich motifs within 100 bp downstream of E2F that are overrepresented among G1/S genes (P = 1.40 E-5 and 8.12 E-5, respectively). Another enriched combination involves E2F and a neighbor motif strongly resembling the binding site of NF-Y (CompareACE score = 0.98). While genes with this combination have a clear preference for peaking in G1/S and S phases (P = 4.39 E-4), it should also be noted that a small number of them belong to G2 and G2/M clusters instead, including CDC2 and CYCLIN B1, whose promoter elements have recently been characterized to contain functional E2F and NF-Y sites (Zhu et al. 2004
Understanding the regulation of the human cell-division cycle is central to the study of many diseases. A recent genome-wide in silico study identified TF binding sites that are overrepresented in the promoters of cell cycle periodic genes (Elkon et al. 2003
Functional interactions between TFs not only require their co-occurrence on the same promoter (enhancer), but often with positional (Makeev et al. 2003
The success of our approach clearly relies on the correct identification of true TF-binding sites from sequences. To estimate the reliability of in silico predictions, we compared the list of E2F site-containing genes with those in vivo targets determined using ChIP-chip technology. While there is a significant overlap between the two, it is worth noting that some ChIP-chip targets are "missed" by sequence-based search. Such apparent discrepancy has been reported before (Iyer et al. 2001
The findings of many significant motif pairs, where neighbor seems to be the same as anchor, underscores the importance of homotypic interactions in transcriptional regulation. Two recent bioinformatics studies have based their search for cis- regulatory modules (CRM) in Drosophila upon the clustering of a single motif (Markstein et al. 2002
Several TF combinations uncovered in our analysis appear to control specific phases of the cell cycle. For example, we found the NF-Y-CDE-CHR module predictive of G2 and G2/M genes. E2F-NF-Y, on the other hand, is preferably associated with those peaking in G1/S and S. NF-Y binding has been reported in many cell cycle promoters (Bolognese et al. 1999 Deciphering transcription regulatory networks from genomic sequence is an exciting but challenging task, especially given the enormous size and complexity of the human genome. In this study, we attempted to uncover the signals that may direct gene expression by searching for evolutionarily conserved and overrepresented TF-binding site combinations associated with more coherent mRNA patterns. While the current analysis was performed with an expression data set obtained from synchronized HeLa cells, it can be readily extended to probe different cellular conditions and types. We anticipate such approaches will be useful for understanding how gene regulation is encoded in the genomic instruction book of life.
Promoter sequences Sequence and human-mouse (HUMMUS) alignment information were obtained from a previous study (Shendure and Church 2002
Anchor motifs
Expression data
Genome-wide scanning for anchor motifs
Search for neighbor motifs
Statistics of overrepresented neighbor motifs
where Nn is the number of promoters with neighbor motif, Nt is the total number of promoters (11436), W is the specified window size (50 or 100 bp), L is the promoter length (1000 bp), Ca is the average copy number of anchor motifs per anchor-containing promoter, and Cn is the average copy number of neighbor motifs per neighbor-containing promoter. The W/L term is included, because a success is scored only if the neighbor motif falls within a particular window around the anchor; the Ca and Cn terms take into account scenarios where there may be more than one copy of anchor or neighbor motif on a promoter, which essentially leads to a larger window size or multiple tests for success, respectively. The probability of getting at least the observed number of (anchor-containing) promoters with neighbor motifs within a specified window by chance follows as:
where Na is the number of promoters with anchor motif, and x is the number of promoters with anchor-neighbor motif combination.
Correcting for multiple hypothesis testing
Identifying functional anchor-neighbor motif combinations
where n = the number of gene pairs in a set, m = the number of closely correlated gene pairs in a set, comb = the set of genes with anchor-neighbor motif combination, neg1 = the set of genes with anchor motif but without neighbor in the vicinity, neg2 = the set of genes with neighbor motif but without anchor in the vicinity, N =nneg1+ncomb+nneg2, and M =mneg1+mcomb+mneg2. We reported the combinations with P-values <7.5 E-5 and 5.7 E-5 for 50 and 100 bp, respectively, as the implied significance level for these cut-offs is 0.05 when applying Bonferroni correction for multiple testing.
Search for genes containing the NF-Y-CDE-CHR module
where x is the number of genes with potential NF-Y-CDE-CHR regulatory module and peak in G2 and G2/M phases, M is the total number of genes that we have both expression data and sequence data for (8162), n is the number of genes with potential NF-Y-CDE-CHR regulatory module, and K is the total number of G2 and G2/M genes (230).
We are grateful to John Aach, Patrik D'haeseleer, Philippe Marc, Allegra Petti, and Fritz Roth for inspiring discussions and valuable comments. We also thank the anonymous reviewers for their helpful feedback. Z.Z. was a Howard Hughes Medical Institute predoctoral fellow. This work was supported by CEGS, DOE-GTL, DARPA, and the Lipper Foundation.
1 Corresponding authors. E-mail zhou-zhu{at}student.hms.harvard.edu; fax (617) 432-7266. E-mail http://arep.med.harvard.edu/gmc/email.html; fax (617) 432-7266. [Supplemental material is available online at www.genome.org and http://genetics.med.harvard.edu/~zhu/combination.html.] Article and publication are at http://www.genome.org/cgi/doi/10.1101/gr.3394405.
Aerts, S., Van Loo, P., Thijs, G., Moreau, Y., and De Moor, B. 2003. Computational detection of cis-regulatory modules. Bioinformatics 19: II5-II14. Arnone, M.I. and Davidson, E.H. 1997. The hardwiring of development: Organization and function of genomic regulatory systems. Development 124: 1851-1864.[Abstract]
Banerjee, N. and Zhang, M.Q. 2003. Identifying cooperativity among transcription factors controlling the cell cycle in yeast. Nucleic Acids Res. 31: 7024-7031. Beer, M.A. and Tavazoie, S. 2004. Predicting gene expression from sequence. Cell 117: 185-198.[CrossRef][Medline] Bertolino, E. and Singh, H. 2002. POU/TBP cooperativity: A mechanism for enhancer action from a distance. Mol. Cell 10: 397-407.[CrossRef][Medline]
Blanchette, M. and Tompa, M. 2002. Discovery of regulatory elements by a computational method for phylogenetic footprinting. Genome Res. 12: 739-748. Bolognese, F., Wasner, M., Dohna, C.L., Gurtner, A., Ronchi, A., Muller, H., Manni, I., Mossner, J., Piaggio, G., Mantovani, R., et al. 1999. The cyclin B2 promoter depends on NF-Y, a trimer whose CCAAT-binding activity is cell-cycle regulated. Oncogene 18: 1845-1853.[CrossRef][Medline] Bussemaker, H.J., Li, H., and Siggia, E.D. 2001. Regulatory element detection using correlation with expression. Nat. Genet. 27: 167-171.[CrossRef][Medline]
Caretti, G., Salsi, V., Vecchi, C., Imbriano, C., and Mantovani, R. 2003. Dynamic recruitment of NF-Y and histone acetyltransferases on cell-cycle promoters. J. Biol. Chem. 278: 30435-30440. Cawley, S., Bekiranov, S., Ng, H.H., Kapranov, P., Sekinger, E.A., Kampa, D., Piccolboni, A., Sementchenko, V., Cheng, J., Williams, A.J., et al. 2004. Unbiased mapping of transcription factor binding sites along human chromosomes 21 and 22 points to widespread regulation of noncoding RNAs. Cell 116: 499-509.[CrossRef][Medline] Chang, H., Shyu, K.G., Lin, S., Tsai, S.C., Wang, B.W., Liu, Y.C., Sung, Y.L., and Lee, C.C. 2003. The plasminogen activator inhibitor-1 gene is induced by cell adhesion through the MEK/ERK pathway. J. Biomed. Sci. 10: 738-745.[CrossRef][Medline] Conkright, M.D., Guzman, E., Flechner, L., Su, A.I., Hogenesch, J.B., and Montminy, M. 2003. Genome-wide analysis of CREB target genes reveals a core promoter requirement for cAMP responsiveness. Mol. Cell 11: 1101-1108.[CrossRef][Medline]
Elkon, R., Linhart, C., Sharan, R., Shamir, R., and Shiloh, Y. 2003. Genome-wide in silico identification of transcriptional regulators controlling the cell cycle in human cells. Genome Res. 13: 773-780.
Euskirchen, G., Royce, T.E., Bertone, P., Martone, R., Rinn, J.L., Nelson, F.K., Sayward, F., Luscombe, N.M., Miller, P., Gerstein, M., et al. 2004. CREB binds to multiple loci on human chromosome 22. Mol. Cell. Biol. 24: 3804-3814. Farina, A., Manni, I., Fontemaggi, G., Tiainen, M., Cenciarelli, C., Bellorini, M., Mantovani, R., Sacchi, A., and Piaggio, G. 1999. Down-regulation of cyclin B1 gene transcription in terminally differentiated skeletal muscle cells is associated with loss of functional CCAAT-binding NF-Y complex. Oncogene 18: 2818-2827.[CrossRef][Medline] Frech, K., Quandt, K., and Werner, T. 1998. Muscle actin genes: A first step towards computational classification of tissue specific promoters. In Silico Biol. 1: 29-38.[Medline]
Garten, Y., Kaplan, S., and Pilpel, Y. 2005. Extraction of transcription regulatory signals from genome-wide DNA-protein interaction data. Nucleic Acids Res. 33: 605-615. Gurdon, J.B. and Bourillot, P.Y. 2001. Morphogen gradient interpretation. Nature 413: 797-803.[CrossRef][Medline]
Hatada, E.N., Chen-Kiang, S., and Scheidereit, C. 2000. Interaction and functional interference of C/EBP
Hertz, G.Z. and Stormo, G.D. 1999. Identifying DNA and protein patterns with statistically significant alignments of multiple sequences. Bioinformatics 15: 563-577. Hirose, T., Sowa, Y., Takahashi, S., Saito, S., Yasuda, C., Shindo, N., Furuichi, K., and Sakai, T. 2003. p53-independent induction of Gadd45 by histone deacetylase inhibitor: Coordinate regulation by transcription factors Oct-1 and NF-Y. Oncogene 22: 7762-7773.[CrossRef][Medline] Hughes, J.D., Estep, P.W., Tavazoie, S., and Church, G.M. 2000. Computational identification of cis-regulatory elements associated with groups of functionally related genes in Saccharomyces cerevisiae. J. Mol. Biol. 296: 1205-1214.[CrossRef][Medline]
Ishida, S., Huang, E., Zuzan, H., Spang, R., Leone, G., West, M., and Nevins, J.R. 2001. Role for E2F in control of both DNA replication and mitotic functions as revealed from DNA microarray analysis. Mol. Cell. Biol. 21: 4684-4699. Iyer, V.R., Horak, C.E., Scafe, C.S., Botstein, D., Snyder, M., and Brown, P.O. 2001. Genomic binding sites of the yeast cell-cycle transcription factors SBF and MBF. Nature 409: 533-538.[CrossRef][Medline] Kato, M., Hata, N., Banerjee, N., Futcher, B., and Zhang, M.Q. 2004. Identifying combinatorial regulation of transcription factors and binding motifs. Genome Biol. 5: R56.[CrossRef][Medline] Kel, A., Kel-Margoulis, O., Babenko, V., and Wingender, E. 1999. Recognition of NFATp/AP-1 composite elements within genes induced upon the activation of immune cells. J. Mol. Biol. 288: 353-376.[CrossRef][Medline] Kel, A.E., Kel-Margoulis, O.V., Farnham, P.J., Bartley, S.M., Wingender, E., and Zhang, M.Q. 2001. Computer-assisted identification of cell cycle-related genes: New targets for E2F transcription factors. J. Mol. Biol. 309: 99-120.[CrossRef][Medline]
Krivan, W. and Wasserman, W.W. 2001. A predictive model for regulatory sequences directing liver-specific transcription. Genome Res. 11: 1559-1566.
Levy, S., Hannenhalli, S., and Workman, C. 2001. Enrichment of regulatory signals in conserved non-coding genomic sequence. Bioinformatics 17: 871-877.
Lifanov, A.P., Makeev, V.J., Nazina, A.G., and Papatsenko, D.A. 2003. Homotypic regulatory clusters in Drosophila. Genome Res. 13: 579-588.
Liu, Y., Liu, X.S., Wei, L., Altman, R.B., and Batzoglou, S. 2004. Eukaryotic regulatory element conservation analysis and identification using comparative genomics. Genome Res. 14: 451-458.
Loots, G.G., Locksley, R.M., Blankespoor, C.M., Wang, Z.E., Miller, W., Rubin, E.M., and Frazer, K.A. 2000. Identification of a coordinate regulator of interleukins 4, 13, and 5 by cross-species sequence comparisons. Science 288: 136-140. Lucibello, F.C., Truss, M., Zwicker, J., Ehlert, F., Beato, M., and Muller, R. 1995. Periodic cdc25C transcription is mediated by a novel cell cycle-regulated repressor element (CDE). EMBO J. 14: 132-142.[Medline]
Makeev, V.J., Lifanov, A.P., Nazina, A.G., and Papatsenko, D.A. 2003. Distance preferences in the arrangement of binding motifs and hierarchical levels in organization of transcription regulatory information. Nucleic Acids Res. 31: 6016-6026. Mantovani, R. 1999. The molecular biology of the CCAAT-binding factor NF-Y. Gene 239: 15-27.[CrossRef][Medline]
Markstein, M., Markstein, P., Markstein, V., and Levine, M.S. 2002. Genome-wide analysis of clustered Dorsal binding sites identifies putative target genes in the Drosophila embryo. Proc. Natl. Acad. Sci. 99: 763-768.
Papatsenko, D.A., Makeev, V.J., Lifanov, A.P., Regnier, M., Nazina, A.G., and Desplan, C. 2002. Extraction of functional binding sites from unique regulatory regions: The Drosophila early developmental enhancers. Genome Res. 12: 470-481. Pilpel, Y., Sudarsanam, P., and Church, G.M. 2001. Identifying regulatory networks by combinatorial analysis of promoter elements. Nat. Genet. 29: 153-159.[CrossRef][Medline]
Ren, B., Cam, H., Takahashi, Y., Volkert, T., Terragni, J., Young, R.A., and Dynlacht, B.D. 2002. E2F integrates cell cycle progression with DNA repair, replication, and G(2)/M checkpoints. Genes & Dev. 16: 245-256. Roth, F.P., Hughes, J.D., Estep, P.W., and Church, G.M. 1998. Finding DNA regulatory motifs within unaligned noncoding sequences clustered by whole-genome mRNA quantitation. Nat. Biotechnol. 16: 939-945.[CrossRef][Medline] Safrany, G. and Perry, R.P. 1995. The relative contributions of various transcription factors to the overall promoter strength of the mouse ribosomal protein L30 gene. Eur. J. Biochem. 230: 1066-1072.[Medline] Shendure, J. and Church, G.M. 2002. Computational discovery of sense-antisense transcription in the human and mouse genomes. Genome Biol. 3: research0044.
Shrivastava, A., Saleque, S., Kalpana, G.V., Artandi, S., Goff, S.P., and Calame, K. 1993. Inhibition of transcriptional regulator Yin-Yang-1 by association with c-Myc. Science 262: 1889-1892. Simpson, P. 2002. Evolution of development in closely related species of flies and worms. Nat. Rev. Genet. 3: 907-917.[CrossRef][Medline]
Storey, J.D. and Tibshirani, R. 2003. Statistical significance for genomewide studies. Proc. Natl. Acad. Sci. 100: 9440-9445.
Tanaka, M., Ueda, A., Kanamori, H., Ideguchi, H., Yang, J., Kitajima, S., and Ishigatsubo, Y. 2002. Cell-cycle-dependent regulation of human aurora A transcription is mediated by periodic repression of E4TF1. J. Biol. Chem. 277: 10719-10726. Tavazoie, S., Hughes, J.D., Campbell, M.J., Cho, R.J., and Church, G.M. 1999. Systematic determination of genetic network architecture. Nat. Genet. 22: 281-285.[CrossRef][Medline]
Terai, G. and Takagi, T. 2004. Predicting rules on organization of cis-regulatory elements, taking the order of elements into account. Bioinformatics 20: 1119-1128.
van Ginkel, P.R., Hsiao, K.M., Schjerven, H., and Farnham, P.J. 1997. E2F-mediated growth regulation requires transcription factor cooperation. J. Biol. Chem. 272: 18367-18374.
Wagner, A. 1999. Genes regulated cooperatively by one or more transcription factors and their identification in whole eukaryotic genomes. Bioinformatics 15: 776-784. Wasserman, W.W. and Fickett, J.W. 1998. Identification of regulatory regions which confer muscle-specific gene expression. J. Mol. Biol. 278: 167-181.[CrossRef][Medline] Wasserman, W.W., Palumbo, M., Thompson, W., Fickett, J.W., and Lawrence, C.E. 2000. Human-mouse genome comparisons to locate regulatory sites. Nat. Genet. 26: 225-228.[CrossRef][Medline]
Weinmann, A.S., Yan, P.S., Oberley, M.J., Huang, T.H., and Farnham, P.J. 2002. Isolating human transcription factor targets by coupling chromatin immunoprecipitation and CpG island microarray analysis. Genes & Dev. 16: 235-244.
Whitfield, M.L., Sherlock, G., Saldanha, A.J., Murray, J.I., Ball, C.A., Alexander, K.E., Matese, J.C., Perou, C.M., Hurt, M.M., Brown, P.O., et al. 2002. Identification of genes periodically expressed in the human cell cycle and their expression in tumors. Mol. Biol. Cell 13: 1977-2000. Wingender, E. 2004. TRANSFAC, TRANSPATH and CYTOMER as starting points for an ontology of regulatory networks. In Silico Biol. 4: 55-61.[Medline]
Yun, J., Chae, H.D., Choy, H.E., Chung, J., Yoo, H.S., Han, M.H., and Shin, D.Y. 1999. p53 negatively regulates cdc2 transcription via the CCAAT-binding NF-Y transcription factor. J. Biol. Chem. 274: 29677-29682. Zhou, Q., Gedrich, R.W., and Engel, D.A. 1995. Transcriptional repression of the c-fos gene by YY1 is mediated by a direct interaction with ATF/CREB. J. Virol. 69: 4323-4330.[Abstract] Zhu, W., Giangrande, P.H., and Nevins, J.R. 2004. E2Fs link the control of G1/S and G2/M transcription. EMBO J. 23: 4615-4626.[CrossRef][Medline] Zwicker, J., Lucibello, F.C., Wolfraim, L.A., Gross, C., Truss, M., Engeland, K., and Muller, R. 1995. Cell cycle regulation of the cyclin A, cdc25C and cdc2 genes is based on a common mechanism of transcriptional repression. EMBO J. 14: 4514-4522.[Medline]
http://club.med.harvard.edu/hummus/hummus.html; Human-mouse sequence conservation. http://faculty.washington.edu/~storey/qvalue/; Q-value package for determining false discovery rate (FDR). http://www.bio.cam.ac.uk/cgi-bin/seqlogo/logo.cgi; Tools for generating sequence logo of motifs.
Received October 21, 2004; accepted in revised format March 31, 2005. This article has been cited by other articles:
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||